DEV Community: Jean-Yves Pellé

Update thousands of servers

Jean-Yves Pellé — Tue, 03 Feb 2026 12:58:53 +0000

Keeping a fleet of servers up to date is one of those tasks that sounds simple until you have a thousand of them. Updating one machine is trivial; updating a thousand, concurrently, with proper error handling, is an infrastructure challenge, whether you're managing Linux, macOS, or Windows hosts. In this tutorial, we'll set up CTFreak to do exactly that: deploy and execute an update script across your entire fleet on a recurring schedule, with a clear view of what succeeded and what didn't. We'll illustrate with Debian-based Linux servers, but the same approach applies to any OS with minimal adaptation.

What we'll build

By the end of this tutorial, you'll have a fully automated update pipeline that handles your entire server fleet. Here's the plan:

Writing a non-interactive update script for Debian-based servers (Debian, Ubuntu, or similar)
Adding an SSH credential to CTFreak
Importing all your nodes from a single YAML file
Creating a project and a scheduled task that targets nodes by tag
Running the task and re-executing on failed nodes

Prerequisites

Before we start, make sure you have:

A fleet of Linux servers to update (we'll use 1,000 in this example). Each server has a hostname following the pattern serverXXXXX.local (from server00001 to server01000), runs a Debian-based distribution, and is accessible via SSH on port 22 with a shared private key and an adminuser account that has passwordless sudo privileges.
A CTFreak instance with an administrator account. CTFreak has no external dependencies, so installation only takes a moment. The Free Edition is sufficient for everything in this tutorial.

Step 1: Writing the update script

The bash script we are going to run on each server is straightforward:

#!/bin/bash
set -euo pipefail

export DEBIAN_FRONTEND=noninteractive

sudo apt-get update -y -q

sudo apt-get upgrade -y -q \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold"

sudo apt-get autoremove -y -q
sudo apt-get autoclean -q

if [[ -f /var/run/reboot-required ]]; then
    if [[ ${CTP_AUTO_REBOOT:-} == "true" ]]; then
        echo "THE SERVER IS GOING TO REBOOT"
        nohup sh -c 'sleep 5 && reboot' &>/dev/null &
        disown
    else
        echo "REBOOT REQUIRED"
    fi
fi

A few things to note:

DEBIAN_FRONTEND=noninteractive prevents apt-get from opening interactive prompts that would block an unattended session.
The two Dpkg::Options flags handle configuration file conflicts automatically: keep the currently installed version (--force-confold) unless the package ships a new file that didn't previously exist (--force-confdef).
Finally, the script checks for /var/run/reboot-required to detect whether a reboot is needed after the upgrade. If the CTP_AUTO_REBOOT environment variable is set to true, the server reboots itself automatically; otherwise it simply logs REBOOT REQUIRED so you can schedule reboots at your convenience. We will define the content of CTP_AUTO_REBOOT later via a dedicated task parameter.

If your fleet includes servers running RPM-based distributions (Rocky Linux, AlmaLinux, RHEL, or similar), replace the apt-get commands with their dnf equivalents.

Step 2: Adding the SSH credential

CTFreak needs a credential to authenticate against your servers over SSH.

Navigate to Credentials and click the New Credential button. Paste your private key, give it a recognizable name like MySSHKey, and save.

You can also use password-based authentication instead of a private key. Both options are supported when creating a credential.

Step 3: Importing nodes from a YAML file

In CTFreak, a node represents a remote server that tasks can target. Nodes are organized into node sources. You could add nodes one by one through the web interface, but with 1,000 servers the practical approach is to create an external node source backed by a YAML file.

Create the file /home/adminuser/ctfreak-nodes.yaml (it must be readable by the OS user running the CTFreak process):


- name: server0001
  tagNames:
    - deb_server
    - linux_server
  osFamily: UNIX
  connectionProtocolType: SSH
  sshConnectionProtocol:
    username: adminuser
    hostname: server0001.local
    port: 22

- name: server0002
  tagNames:
    - deb_server
    - linux_server
  osFamily: UNIX
  connectionProtocolType: SSH
  sshConnectionProtocol:
    username: adminuser
    hostname: server0002.local
    port: 22

## ...
## Complete with nodes 3 to 1000
## ...

Notice the tagNames field. Tags let you organize nodes into logical groups and, more importantly, target them from tasks. We'll use the deb_server tag in the next step to tell CTFreak which nodes should receive the update.

Navigate to Nodes and click the New External Node Source button.

Select MySSHKey as the Credential (CTFreak will use it when connecting to these nodes via SSH), then save. The new node source, named Linux Servers in this example, will automatically resynchronize its nodes from the YAML file every two hours.

If the file was processed successfully, you should see all your nodes listed:

Step 4: Creating the project and task

Navigate to Projects and click the New Project button. Name your project Sysadmin (or any name that fits your organization), then save.

Once you are on the project page, click the New Task button, choose Bash Script as the task type, and configure the task with the following settings:

Name: Upgrade debian servers
Parameters: add a checkbox parameter named AUTO_REBOOT (default unchecked)
Bash script: paste the update script from Step 1
Node Set filter: #deb_server (targets every node carrying this tag)
Schedule: a cron expression such as 0 4 1 * * (4:00 AM on the 1st of every month)

Save the task.

Because we used a tag-based filter rather than an explicit list of hostnames, any new node you add to the YAML file with the deb_server tag will automatically be included in future executions, with no task reconfiguration needed.

Step 5: Running the task and handling failures

Rather than waiting for the first of the month, let's trigger a manual run. Click the Execute button.

As you can see, you can specify whether or not you want to automatically reboot servers that need it after the update (in the case of a scheduled launch, the default value AUTO_REBOOT = false will be applied).

The execution starts and CTFreak connects to the matching nodes concurrently (sequentially in the Free Edition). You can watch progress on a per-node basis.

When the execution completes, some nodes will almost certainly have failed. This is expected at scale: an SSH connection may time out, a package mirror might be temporarily unreachable, or a DNS resolution could fail. The important thing is that you have clear visibility into which nodes failed and why.

Check the Failed filter to isolate the problematic nodes and inspect their logs. Once you've addressed the underlying issue (or simply want to retry transient failures), click Re-execute failed nodes. CTFreak will run the same task again, but only against the nodes that failed in the previous execution. This saves time and avoids redundant work on the nodes that already succeeded.

Going further

Once the basic pipeline is running, there are several ways to extend it:

Notifications: attach an email or Slack notifier to the task so your team is alerted immediately when an update run finishes with failures.
Mixed-OS workflows: add RPM-based Linux nodes (Rocky Linux, AlmaLinux, RHEL) with a rpm_server tag and Windows nodes with a windows_server tag. Create a dedicated task for each OS (using Bash for Unix, PowerShell for Windows) and tie them all together in a workflow task.
Ansible playbooks: replace the Bash or PowerShell script with an Ansible playbook to leverage built-in idempotency. Ansible modules like apt and dnf handle package state declaratively, so you don't have to manage idempotence yourself.
Concurrency tuning: adjust the number of workers (aka Max number of concurrent node executions) in the task configuration to control how many nodes are updated in parallel. This is useful if your network or package mirrors can't handle a thousand simultaneous connections.

Wrapping up

With a single YAML file, one SSH credential, and a scheduled task, CTFreak gives you a repeatable, observable process for keeping your entire server fleet up to date. Failed nodes are clearly surfaced and can be retried with one click, which turns a potentially chaotic maintenance window into a routine operation.

The same pattern applies well beyond OS updates: configuration changes, log collection, security audits, or any script you need to run across many machines. Define the nodes, write the script, schedule the task, and let CTFreak handle the concurrency and error tracking.

Happy automating!

Run a shell script from a webhook call

Jean-Yves Pellé — Wed, 12 Nov 2025 09:55:33 +0000

If you've ever found yourself thinking I wish I could just run this script via curl, then webhooks are about to become your new best friend. In this tutorial, we'll explore how to transform CTFreak into a powerful webhook server that can trigger your automation tasks with nothing more than an HTTP request.

The beauty of this approach? You can integrate your scripts and playbooks into virtually any workflow, whether it's responding to GitHub events, processing form submissions, or reacting to monitoring alerts. Let's dive in and build something practical.

What we'll build

By the end of this tutorial, you'll have a fully functional webhook endpoint that executes a Bash script with parameters passed directly through the URL. We'll then show you how this same pattern works seamlessly with Powershell scripts and Ansible playbooks.

Here's what we'll cover:

Adding a node to CTFreak with SSH authentication
Creating a parameterized Bash script task
Setting up an incoming webhook
Triggering the script via HTTP with query parameters
Verifying execution through the logs

Prerequisites

Before we start, make sure you have:

A CTFreak instance up and running
A Unix server accessible via SSH where your scripts will run
An SSH key for connecting to the server
Admin access to your CTFreak instance

Step 1: Adding your node

First things first, we need to tell CTFreak about the server where your scripts will execute.

Navigate to Credentials → New Credential and add your SSH key. Make sure to paste your private key in the appropriate field and give it a memorable name, something like Production server key rather than That key I found in my downloads folder.

Once your credential is saved, head over to Nodes → Internal nodes → New node to register your server. Fill in the following details:

Name: A descriptive name for your server (e.g., webhook-runner)
OS Family: Choose Unix for Linux servers
Username: The SSH user CTFreak will use to connect
Hostname: Your server's IP address or hostname
Port: SSH port, typically 22
Credential: Select the SSH key you just created

Click Create, then verify that the connection to your node is working by attempting to open the file explorer (via the Browse button). If you see the contents of a directory, congratulations, you're ready to move forward.

Step 2: Creating a parameterized Bash script task

Now comes the fun part. We're going to create a task that accepts parameters, which means the same webhook can behave differently based on the values you pass to it. This is where the real magic happens.

Navigate to Projects → New project if you don't already have a suitable project, or use an existing one. Let's call our project Webhook Automation for clarity.

Inside your project, click New task and select Bash Script as the task type. You'll see a form with several fields to fill out:

Name: Give your task a descriptive name like Process deployment webhook

Script: This is where you write your Bash script. For this example, we'll create a simple script that uses two parameters:

#!/bin/bash

echo "Webhook triggered with parameters:"
echo "Environment: $CTP_ENVIRONMENT"
echo "Version: $CTP_VERSION"

# Your actual deployment logic would go here
# For example:
# cd /var/www/app
# git pull origin $CTP_VERSION
# systemctl restart myapp-$CTP_ENVIRONMENT

Notice how we're using environment variables prefixed with CTP_. This is CTFreak's convention for injecting parameters into your scripts, and it works beautifully once you get used to it.

Node Set filter: Enter the node name we created earlier.

Now, let's add our parameters. Click the Add parameter button twice to create two parameters:

First parameter:

Name: ENVIRONMENT
Parameter type: Selector
Options: Add values like "staging", "production"
Default value: "staging"

Second parameter:

Name: VERSION
Parameter type: Text
Default value: "main"

The beauty of this setup is that you can make the parameters as restrictive or flexible as you need. Selector types limit choices to prevent mishaps (nobody wants to accidentally deploy to production when they meant staging), while text types give you complete freedom.

Save your task, and let's move on to exposing it to the world via a webhook.

Step 3: Creating the incoming webhook

This is where CTFreak transforms from a task automation tool into a full-fledged webhook server. From the task we just created, head to Incoming webhooks → New webhook → Generic.

You'll need to configure a few things:

Name: Something descriptive like "Deployment trigger"

Enabled: Make sure this toggle is on, or you'll be wondering why nothing happens when you call it.

Save the webhook, and CTFreak will display your webhook URL along with the authentication details. It'll look something like:

https://your-ctfreak-instance/wh/VEihowf6DHiDj1C2iZtXjTI7alFxFXzVoSareJeD7EskYzx2t1InyBEB6Wuq

Copy this URL, you're about to use it.

Step 4: Triggering the webhook

Now for the moment of truth. You can trigger your webhook from anywhere that can make an HTTP request: a CI/CD pipeline, a monitoring tool, another script, or just your terminal for testing.

Here's how to call it with curl, passing our two parameters as query strings:

curl -X POST "https://your-ctfreak-instance/wh/VEihowf6DHiDj1C2iZtXjTI7alFxFXzVoSareJeD7EskYzx2t1InyBEB6Wuq?ENVIRONMENT=production&VERSION=v2.1.0"

Notice how we're passing ENVIRONMENT and VERSION as query parameters? CTFreak automatically maps these to the corresponding task parameters and injects them as environment variables (prefixed with CTP-) into your script.

You can also pass parameters as URL-encoded form submission if you prefer:

curl -X POST "https://your-ctfreak-instance/wh/VEihowf6DHiDj1C2iZtXjTI7alFxFXzVoSareJeD7EskYzx2t1InyBEB6Wuq" \
-d "ENVIRONMENT=production" \
-d "VERSION=v2.1.0"

Both methods work equally well, so use whichever fits your workflow better.

The webhook will return a response immediately, including an execution ID that you can use to check the status and logs later:

{"status":200,"executionId":"01K9STQAFNQ8MKB47F4DZ32RH7","message":"I got the payload!"}

This is perfect for asynchronous workflows where you fire off a task and check back later to see how it went.

Step 5: Verifying the execution

Let's make sure everything worked as expected. Navigate to Projects → Webhook Automation → Process deployment webhook. You should see your webhook-triggered execution at the top of the execution list.

Click on the execution ID, and on the eye icon to view the execution logs. You'll see the output from your script, including the parameter values that were passed in:

Beautiful. Your parameters made it through the HTTP request, into CTFreak, and into your script exactly as intended. This is the foundation you can build upon for much more complex workflows.

Beyond Bash: Powershell and Ansible

Here's where things get even better. The webhook pattern we just built works identically with other task types. Let me show you how.

Powershell scripts

If you're working with Windows servers, simply create a Powershell Script task instead of Bash Script task. When adding your node, set the OS Family to Windows and adjust your credentials according to the chosen connection protocol (SSH or WinRM).

Your script parameters work exactly the same way, just with Powershell syntax:

Write-Host "Webhook triggered with parameters:"
Write-Host "Environment: $env:CTP_ENVIRONMENT"
Write-Host "Version: $env:CTP_VERSION"

# Your Powershell deployment logic here

The webhook setup? Identical. The curl command? Identical. The only thing that changes is the language of your script itself.

Ansible playbooks

Running Ansible playbooks via webhooks is just as straightforward. Configure a task of type Ansible Playbook and add your parameters.

CTFreak will inject the parameters as extra variables, which you can access in your playbook like this:

- hosts: all
  tasks:
    - name: Display received parameters
      debug:
        msg: "Deploying version {{ CTP_VERSION }} to {{ CTP_ENVIRONMENT }}"

    - name: Your deployment workflow here
      # ...

Same webhook URL, same query parameters. The power of this approach is that once you understand the pattern, you can apply it to any task type CTFreak supports.

Real-world use cases

Now that you've got the basics down, let's talk about what you can actually do with this. Here are a few ideas to spark your imagination:

CI/CD integration: Trigger deployments from GitHub Actions, GitLab CI, or Jenkins after successful builds. Pass the branch name, commit hash, and environment as parameters.

Infrastructure automation: Respond to monitoring alerts by automatically scaling resources, restarting services, or running diagnostic scripts. Your monitoring tool hits a webhook, CTFreak handles the remediation.

ChatOps: Integrate with Slack or Microsoft Teams to let your team trigger common operations through chat commands. Someone types "/deploy staging v2.0" and behind the scenes, it's just a webhook call.

IoT and edge computing: Have your devices report status by hitting a webhook that triggers data processing scripts or updates dashboards.

Business process automation: Connect forms, CRM systems, or e-commerce platforms to trigger backend processes when specific events occur.

Wrapping up

You've now got a powerful webhook server that can execute arbitrary scripts with parameters, and you didn't have to write a single line of server code to make it happen. CTFreak handles the infrastructure, logging, and parameter injection, so you can focus on writing the scripts that actually do useful work.

The pattern we've explored here is deceptively simple, but it opens up endless possibilities for automation and integration. Whether you're orchestrating deployments, automating infrastructure tasks, or building custom integrations, webhooks give you a clean, HTTP-based interface to trigger your operations from anywhere.

Happy automating!

Deploy a lightweight BI solution with your first dashboard in 5 steps

Jean-Yves Pellé — Sun, 01 Sep 2024 09:19:08 +0000

This article is aimed at those who know SQL and want to quickly set up a minimalist reporting solution (without going through the heavy artillery of PowerBI, Tableau, Looker Studio, etc.).

Let's start with a database (of the Mysql, Mariadb or Postgresql type) including a monthly_sales table containing monthly sales amounts:

month	sales_amount_eur
`2024-01-01`	`8730`
`2024-02-01`	`9620`
`2024-03-01`	`4210`
`2024-04-01`	`6732`
`2024-05-01`	`9921`
`2024-06-01`	`8176`
`2024-07-01`	`7623`

Our aim is to:

Generate a report showing this information in a graph.
Refresh it every 1st of the month.

1 - Installation

Let's start by installing CTFreak.

Under Ubuntu, open your terminal and run:

sudo snap install ctfreak

For alternative installations (Docker, Windows, Freebsd, ...), see here.

2 - Login

Go to http://localhost:6700 and log in via admin / ctfreak.

3 - Adding a database

Go to Databases → New Database, select its type (e.g. Postgresql), then fill in its connection parameters:

Validate to add the database.

4 - Creating a project

A project will group together all our reporting tasks.

Go to Projects → New Project:

Validate to create the project.

5 - Creating a SQL Report task

Our report will be generated via a SQL Report task associated with our project.

To do this, go to Projects → Reporting → New Task, select SQL Report as the task type and then fill in the following information:

In the SQL query, the suffix _c_month allows you to indicate that date values should be formatted as months in the graph.

Validate this form to create a task which, based on an SQL query, will generate a monthly sales report every 1st of the month at 9 a.m., with a 2 year (730 days) retention period for the reports generated.

Let's execute this task right now (via Projects → Reporting → Sales report -> Execute) to get the expected report:

Conclusion

As you can see, this report meets our initial requirements.

Feel free to use CTFreak's other features to make improvements, such as:

Add more charts.
Add task parameters to define the sales period to be taken into account.
Add read-only access for a given user (via the Viewer role).
Send an e-mail each time a report is generated, with a link to open it (via a Notifier).
Synchronize the generation of multiple reports (via Workflow tasks).

Déployer une solution de BI légère avec son premier dashboard en 5 étapes

Jean-Yves Pellé — Sat, 31 Aug 2024 14:04:37 +0000

Cet article s'adresse à ceux qui connaissent le SQL et souhaitent mettre en place rapidement une solution de reporting minimaliste (sans passer par l'artillerie lourde comme PowerBI, Tableau, Looker Studio, et consorts).

Partons d'une base de données (de type Mysql, Mariadb ou Postgresql) comprenant une table monthly_sales contenant des montants mensuels de vente:

month	sales_amount_eur
`2024-01-01`	`8730`
`2024-02-01`	`9620`
`2024-03-01`	`4210`
`2024-04-01`	`6732`
`2024-05-01`	`9921`
`2024-06-01`	`8176`
`2024-07-01`	`7623`

Notre but est de:

Générer un rapport reprenant ces informations dans un graphique.
Le rafraîchir tous les 1er du mois.

1 - Installation

Commençons par installer CTFreak.

Sous Ubuntu, ouvrez votre terminal et lancez:

sudo snap install ctfreak

Pour les installations alternatives (Docker, Windows, Freebsd, ...), ça se passe ici.

2 - Connexion

Rendez vous sur http://localhost:6700 et connectez-vous via admin / ctfreak.

3 - Ajout d'une base de données

Allez dans Databases → New Database, sélectionnez son type (par exemple Postgresql), puis renseignez ses paramètres de connexion:

Validez pour ajouter la base de données.

4 - Création d'un projet

Un projet va permettre de regrouper toutes nos tâches de reporting.

Allez dans Projects → New Project:

Validez pour créer le projet.

5 - Création d'une tâche SQL Report

Notre rapport sera généré via une tâche de type SQL Report associé à notre projet.

Pour ce faire, allez dans Projects → Reporting → New Task, sélectionnez SQL Report comme type de tâche puis renseignez les informations suivantes:

Dans la requête SQL, le suffixe _c_month permet d'indiquer que les valeurs de date doivent être formatées comme des mois dans le graphique.

Validez ce formulaire pour créer une tâche qui à partir d'une requête SQL générera un rapport des ventes mensuelles tous les 1er du mois à 9h, avec une rétention de 2 ans (730 jours) des rapports générés.

Exécutons cette tâche dès maintenant (via Projects → Reporting → Sales report -> Execute) pour obtenir le rapport attendu:

Conclusion

Comme vous pouvez le constater, ce rapport répond bien à notre besoin initial.

N'hésitez pas à exploiter les autres fonctionnalités de CTFreak pour y apporter quelques améliorations comme:

Ajouter d'autres graphiques.
Ajouter des paramètres de tâche pour définir la période des ventes à prendre en compte.
Ajouter un accès en lecture seule pour un utilisateur donné (via le rôle Viewer).
Envoyer à chaque génération du rapport un mail avec un lien pour l'ouvrir (via un Notifier).
Synchroniser la génération de plusieurs rapports (via les tâches de type Workflow).

A web form to launch your bash and powershell scripts

Jean-Yves Pellé — Mon, 18 Mar 2024 11:23:41 +0000

What's the point of using a web interface to run scripts when you're a terminal master?

Well, among other things:

To delegate the execution of certain scripts to business users securely, by limiting the available options to prevent any unpleasant surprises and keeping a record of executions:

Exemple: Generate a report for a given date range and send it by email
Or to easily execute predefined tasks in case of emergency (a terminal is not always at hand, unlike a smartphone):

Exemple: Check the status or restart/stop a Linux service

I propose addressing these two use cases using the newly added parameterizable tasks in the latest release of Ctfreak.

Prerequisites

We need:

A Unix server: Linux, Mac OS X, FreeBSD, ... (see end of article to adapt to a Windows server) accessible via SSH where your scripts will be executed. We'll call it scriptsrv.
An instance of Ctfreak with an admin account (the Free Edition will suffice if you adapt the examples in this article by limiting to one parameter per task).

Configuration of the scriptsrv server

Connect to your Ctfreak instance with an admin account.

Start by adding the SSH key to connect to the server via SSH Credential → New SSH Credential:

Then add the server itself via Nodes → Internal Nodes → New node:

Make sure that the SSH Key for scriptsrv is selected as the Credential, then validate to create the scriptsrv node.

Ctfreak can now connect to the server with the specified SSH key to execute various tasks.

1st case: Generate a report for a given date range and send it by email

For our example, let's assume that you have an executable generatereport on the scriptsrv server, allowing you to generate and send a sales report by email for a given period, recipient, and format. If we were to run this executable directly in a terminal, it might look something like this:

generatereport --from 2024-03-04 --to 2024-03-10 --recipient nigel@mycompany.zzz --format pdf

So, we'll create a task in Ctfreak that, when executed, will dynamically generate a web form to enter these 4 parameters before launching the generatereport executable.

Creating the Reporting project

Let's start by creating a project dedicated to reporting tasks.

Go to Projects → New Project

Validate to create the project.

Creating the task Generate and send sales report

Go to Projects → Reporting → New task

Choose Bash Script as the task type.

Fill in the Name, Script, and Node filter fields as indicated.

In the content of our script, the 4 expected values are provided through environment variables, all prefixed by CTP_:

#!/bin/bash
generatereport --from $CTP_FROM --to $CTP_TO --recipient $CTP_RECIPIENT --format $CTP_FORMAT

These variables correspond to the parameters that we will create by clicking the Add parameter button.

Let's start with the start and end dates of our period:

The parameter type Date ensures that a valid date will be entered in our execution form.
The Default date type field allows us to take the previous week as the default period.
The Name field will be used to define the name of the environment variable (CTP_FROM in our script thus refers to the parameter named FROM).
When the Label field is filled, it replaces the Name field to reference the variable during the execution form generation.
During task execution, the entered date will be converted to the YYYY-MM-DD format to populate the corresponding environment variable.

Regarding the emails of authorized recipients to receive the report, we will limit the possible choices to 2 emails via the Selector parameter type:

For each option, when the Label field is filled, it substitutes Value during the execution form generation. We could have used it here to display Nigel and John in the form instead of their respective emails.

Finally, another Selector parameter type will allow choosing the format of the generated file (here PDF or Excel, but it's pdf or xlsx that will be passed to our script):

With all the necessary information filled in, all that's left is to validate to create the task.

Executing the task

Go to Projects → Reporting → Generate and send sales report → Execute to bring up the execution form.

As you can see, our 4 parameters are well taken into account in the form generation.

Adding a business user

With our task operational, the next step is to hand it over to a business user for execution.

Go to Users → New User

Validate to create the user.

By default, a non-admin user has no access to anything (except for logging in), so you need to give them the necessary rights.

Go to Projects → Reporting → Access → Edit

Add Jack by giving him the Executor role.

Jack now has access to the Reporting project, but only to execute the tasks it contains.

2nd case: Check the status or restart / stop a Linux service

Creating the Sysadmin project

Let's create a project dedicated to system administration tasks.

Go to Projects → New Project

Validate to create the project.

Creating the task Systemctl

Go to Projects → Sysadmin → New task and like our previous case, choose Bash Script as the task type.

Fill in the Name, Script, and Node filter fields as indicated.

Click the Add parameter button to add the list of services concerned (here Postgresql and Apache)

As well as the commands to check the status, restart, or stop the service.

Validate to create the task.

Executing the task

Go to Projects → Sysadmin → Systemctl → Execute to bring up the execution form.

Execute the task and check the logs by clicking on the eye icon.

Running Powershell scripts instead of Bash

If your scriptsrv server is running Windows, you just need to make a few adjustments to run Powershell scripts instead of Bash:

Activate the internal SSH server in Windows.
When creating the scriptsrv node, set the OS Family property to Windows instead of Unix.
Create tasks of type Powershell Script instead of Bash Script and write the content of your scripts in Powershell instead of Bash.

The principle of injecting parameters as environment variables remains exactly the same. If we adapted the script from our first case, it would look like this:

generatereport --from $env:CTP_FROM --to $env:CTP_TO --recipient $env:CTP_RECIPIENT --format $env:CTP_FORMAT

Conclusion

As you have noticed, simply adding a web interface to a script offers many possibilities.

Start by adapting the examples presented in this article to your needs; they will already allow you to offer greater autonomy to your users.

Un formulaire web pour lancer vos scripts bash et powershell

Jean-Yves Pellé — Sun, 17 Mar 2024 11:03:33 +0000

A quoi bon passer par une interface web pour lancer des scripts lorsque vous êtes un pro du terminal ?

Et bien entre autres:

Pour déléguer aux utilisateurs métiers l'exécution de certains scripts de manière sécurisée, en limitant les options disponibles afin de prévenir toute mauvaise surprise et en gardant l'historique des exécutions:

Exemple: Générer un rapport pour une plage de dates donnée et l'envoyer par mail
Ou pour exécuter facilement des tâches prédéfinies en cas d'urgence (un terminal n'est pas toujours à portée de main, contrairement à un smartphone):

Exemple: Connaitre l’état ou redémarrer / arrêter un service linux

Je vous propose de répondre à ces 2 cas d'usage via l'utilisation des tâches paramétrables qui viennent d'être ajoutées dans la dernière release de Ctfreak.

Prérequis

Il nous faut:

Un serveur Unix: Linux, Mac OS X, FreeBSD, ... (voir en fin d'article pour adapter à un serveur Windows) accessible via SSH où seront exécutés vos scripts. Nous le nommerons scriptsrv.
Une instance de Ctfreak avec un compte administrateur (la Free Edition suffira si vous adaptez les exemples de cet article en vous limitant à un paramètre par tâche).

Configuration du serveur scriptsrv

Connectez vous à votre instance Ctfreak avec un compte administrateur.

Commencez par ajouter la clef SSH permettant de se connecter au serveur via SSH Credential → New SSH Credential:

Puis ajoutez le serveur en lui même via Nodes → Internal Nodes → New node:

Vérifiez que la clef SSH Key for scriptsrv est bien sélectionnée comme Credential, puis validez pour créer le node scriptsrv.

Ctfreak peut désormais de connecter au serveur avec la clef SSH spécifiée pour y exécuter différentes tâches.

1er cas: Générer un rapport pour une plage de dates donnée et l'envoyer par mail

Pour notre exemple, nous allons partir du principe que vous disposez sur le serveur scriptsrv d'un exécutable generatereport permettant de générer et d'envoyer par mail un rapport des ventes pour une période, un destinataire et un format donné. Si nous devions lancer cet exécutable directement dans un terminal, cela pourrait donner quelque chose comme:

generatereport --from 2024-03-04 --to 2024-03-10 --recipient nigel@mycompany.zzz --format pdf

Nous allons donc créer dans Ctfreak une tâche qui lors de son exécution génèrera à la volée un formulaire web permettant de renseigner ces 4 paramètres avant de lancer l'exécutable generatereport.

Création du projet Reporting

Pour commencer, créons un projet dédié aux tâches de reporting.

Allez dans Projects → New Project

Validez pour créer le projet.

Création de la tâche Generate and send sales report

Allez dans Projects → Reporting → New task

Choisissez Bash Script comme type de tâche.

Renseignez comme indiqué les champs Name, Script et Node filter.

Dans le contenu de notre script, les 4 valeurs attendues sont renseignées via des variables d'environnement, toutes préfixées par CTP_:

#!/bin/bash
generatereport --from $CTP_FROM --to $CTP_TO --recipient $CTP_RECIPIENT --format $CTP_FORMAT

Ces variables correspondent aux paramètres que nous allons créer en cliquant sur le bouton Add parameter.

Commençons par les dates de début et de fin de notre période:

Le type de paramètre Date permet de s'assurer qu'une date valide sera saisie dans notre formulaire d'exécution.
Le champ Default date type permet de prendre la semaine précédente comme période par défaut.
Le champs Name sera utilisé pour définir le nom de la variable d'environnement (CTP_FROM dans notre script fait donc référence au paramètre nommé FROM).
Lorsque le champ Label est renseigné, il remplace le champ Name pour faire référence à la variable lors de la génération du formulaire d'exécution.
Lors de l'exécution de la tâche, la date saisie sera convertie au format YYYY-MM-DD pour alimenter la variable d'environnement correspondante.

En ce qui concerne les mails des destinataires autorisés à recevoir le rapport, nous limiterons les choix possibles à 2 mails via le type de paramètre Selector:

Pour chaque option, le champs Label lorsqu'il est renseigné se substitue à Value lors de la génération du formulaire d'exécution. Nous aurions pu l'utiliser ici pour afficher Nigel et John dans le formulaire au lieu de leurs emails respectifs.

Pour terminer, un dernier paramètre de type Selector permettra de choisir le format du fichier généré (ici PDF ou Excel, mais c'est bien pdf ou xlsx qui sera transmis à notre script):

Toutes les informations nécessaires étant renseignées, il ne reste plus qu'à valider pour créer la tâche.

Exécution de la tâche

Allez dans Projects → Reporting → Generate and send sales report → Execute pour faire apparaitre le formulaire d'exécution.

Comme vous pouvez le constater, nos 4 paramètres sont bien pris en compte dans la génération du formulaire.

Ajout d'un utilisateur métier

Notre tâche étant opérationnelle, reste à donner la main à un utilisateur métier afin qu'il puisse l'exécuter.

Allez dans Users → New User

Validez pour créer l'utilisateur.

De base, un utilisateur non-admin n'a accès à rien (mis à part se connecter), il faut donc lui ajouter les droits nécessaires.

Allez dans Projects → Reporting → Access → Edit

Ajouter Jack en lui donnant le rôle Executor.

Jack a désormais accès au projet Reporting, mais uniquement pour exécuter les tâches qu'il contient.

2ème cas: Connaitre l’état ou redémarrer / arrêter un service linux

Création du projet Sysadmin

Créons un projet dédié aux tâches d'administration système.

Allez dans Projects → New Project

Validez pour créer le projet.

Création de la tâche Systemctl

Allez dans Projects → Sysadmin → New task et comme pour notre cas précédent choisissez Bash Script comme type de tâche.

Renseignez comme indiqué les champs Name, Script et Node filter.

Cliquez sur le bouton Add parameter pour ajouter la liste des services concernés (ici Postgresql et Apache)

Ainsi que les commandes pour connaitre l’état, redémarrer ou arrêter le service.

Validez pour créer la tâche.

Exécution de la tâche

Allez dans Projects → Sysadmin → Systemctl → Execute pour faire apparaitre le formulaire d'exécution.

Exécutez la tâche et consultez les logs en cliquant sur l’icône en forme d’œil.

Lancer des scripts Powershell plutôt que Bash

Si votre serveur scriptsrv est sous Windows, il suffit de faire quelques ajustements pour lancer des scripts Powershell plutôt que Bash:

Activez le serveur SSH interne de Windows.
Lors de la création du node scriptsrv, définissez la propriété OS Family à Windows plutôt que Unix.
Créez des tâches de type Powershell Script au lieu de Bash Script et écrivez le contenu de vos scripts en Powershell plutôt que Bash.

Le principe d'injection des paramètres sous forme de variables d'environnement reste exactement le même. Si nous adaptions le script de notre premier cas, cela donnerait:

generatereport --from $env:CTP_FROM --to $env:CTP_TO --recipient $env:CTP_RECIPIENT --format $env:CTP_FORMAT

Pour finir

Comme vous l'aurez constaté, le simple fait d'apposer une interface web sur un script offre beaucoup de possibilités.

Commencez par adapter les exemples présentés dans cet article à vos besoins, ils vous permettrons déjà d'offrir une plus grande autonomie à vos utilisateurs.

Configuring Apache Superset 3 in a production environment

Jean-Yves Pellé — Wed, 11 Oct 2023 10:20:19 +0000

Apache Superset proves to be a fantastic Business Intelligence tool: open-source, feature-rich (with a multitude of charts, integration with multiple database management systems, etc.), and it has nothing to envy in comparison to well-known commercial alternatives (Tableau, PowerBI, BusinessObjects, and the like).

Unfortunately, being fantastic doesn't mean it's without flaws, particularly in terms of documentation, which may seem a bit sparse to some.

It is precisely to address this shortcoming that I propose we look at how to quickly set up a Superset instance tailored for production together.

Prerequisites

Have a production server running Linux (regardless of the distribution) with:
- docker
- git
- caddy (or another reverse proxy of your choice)
- ports 443 and 80 open at the firewall level
A domain name (e.g., bi.myawesomecompany.com) that points to your server

Installation

Cloning the superset repo

Start by cloning the repository directly on your production server (yes, this is not common):

cd $HOME
git clone https://github.com/apache/superset.git
cd superset

Then select the desired version via its tag.

git checkout tags/3.0.0

Editing configuration files

Now comes the step of customizing certain configuration files. Since you are on a git tag, you won't be able to commit your modifications as is, so you have the choice of:

documenting all your modifications to reproduce them exactly in case of reinstallation
or forking the project and creating a branch to commit your modifications

It's up to you.

The docker/.env-non-dev file (understand here: non-dev = prod) allows you to define a set of environment variables that will be used by the docker containers we will start later.

Add the following:

# We don't want demo data in production
SUPERSET_LOAD_EXAMPLES=no
# A random string to encode session cookies
SUPERSET_SECRET_KEY=4Sido8BkIjs54Vz2XyVD5GJIvANVIAT399dRESjdmr4vm92n
# To prevent XSS attacks (among other things)
TALISMAN_ENABLED=yes
# Number of workers: the higher the value, the fewer intermittent chart refresh failures you will have in your dashboards (adjust according to your server's power).
SERVER_WORKER_AMOUNT=64

Also, make some adjustments in docker/pythonpath_dev/superset_config.py to enable alerting and the template engine (necessary for creating datasets with dynamic filtering):

FEATURE_FLAGS = {"ALERT_REPORTS": True, "ENABLE_TEMPLATE_PROCESSING": True}

Disable telemetry by replacing in the docker-compose-non-dev.yml file:

x-superset-image: &superset-image apachesuperset.docker.scarf.sh/apache/superset:${TAG:-latest-dev}

with

x-superset-image: &superset-image apache/superset:${TAG:-latest-dev}

And switch to the latest stable version of postgreSQL by replacing:

     image: postgres:14

with

     image: postgres:16

Startup

Instantiate and start the docker containers:

docker compose -f docker-compose-non-dev.yml up -d

Superset is accessible on your production server via http://127.0.0.1:8088.

Reverse proxy

Configure a reverse proxy to secure the connection, for example, using caddy:

Edit /etc/caddy/Caddyfile

bi.myawesomecompany.com {
        reverse_proxy http://127.0.0.1:8088
}

Then restart caddy:

sudo service caddy restart

First Login

Change your password.

Backup and Restoration

When editing the docker-compose-non-dev.yml configuration file, you may have noticed that a postgresql database is being instantiated.

Therefore, backup and restoration for superset only need to consider this database.

You can perform a hot backup with a simple command:

docker exec -t superset_db pg_dump superset -U superset | xz > backup.sql.xz

For restoration, start only the postgresql container in advance, avoiding having a superset instance connected to a database being restored:

docker compose down
docker compose -f docker-compose-non-dev.yml up db -d
docker exec -t superset_db dropdb -U superset superset
docker exec -t superset_db createdb -U superset superset
xz -dc backup.sql.xz | docker exec -i superset_db psql -U superset -d superset
docker compose -f docker-compose-non-dev.yml up -d

Configurer Apache Superset 3 dans un environnement de production

Jean-Yves Pellé — Mon, 09 Oct 2023 12:04:52 +0000

Apache Superset se révèle être un formidable outil de Business Intelligence: open source, riche en fonctionnalités (multitude de charts, intégration avec plusieurs SGBD, ...) et qui n'a rien à envier aux alternatives commerciales les plus connues (Tableau, PowerBI, BusinessObjects et consorts).

Formidable ne veut malheureusement pas dire qu'il est exempt de défauts, notamment sa documentation qui pourra paraître un peu frugale à certains.

C'est justement pour palier à ce manque que je vous propose de voir ensemble comment mettre en place rapidement une instance de Superset taillée pour la production.

Prérequis

Disposer d'un serveur de production sous linux (peu importe la distribution) avec:
- docker
- git
- caddy (ou autre reverse proxy de votre choix)
- les ports 443 et 80 ouverts au niveau du firewall
Un nom de domaine (par exemple bi.myawesomecompany.com) qui pointe vers votre serveur

Installation

Clonage du repo superset

Commencez par faire un git clone du repo directement sur votre serveur de prod (oui ça n'est pas commun):

cd $HOME
git clone https://github.com/apache/superset.git
cd superset

Puis sélectionnez la version souhaitée via son tag.

git checkout tags/3.0.0

Édition des fichiers de configuration

Arrive l’étape de customisation de certains fichiers de configuration. Comme vous êtes sur un tag git, vous ne pourrez pas commiter vos modifications en l'état, vous aurez donc le choix de:

documenter toutes vos modifications pour les reproduire à l'identique en cas de réinstallation
ou forker le projet et créer une branche pour commiter vos modifications

A vous de voir.

Le fichier docker/.env-non-dev (comprendre ici: non-dev = prod) permet de définir un ensemble de variables d'environnement qui seront utilisées par les containers dockers que nous démarrerons par la suite.

Ajoutez-y ceci:

# On ne veut pas de données de démonstration en prod
SUPERSET_LOAD_EXAMPLES=no
# Une chaîne de caractère aléatoire pour encoder les cookies de session
SUPERSET_SECRET_KEY=4Sido8BkIjs54Vz2XyVD5GJIvANVIAT399dRESjdmr4vm92n
# Pour prévenir les attaques de type XSS (entre autre)
TALISMAN_ENABLED=yes
# Nombre de workers: plus la valeur est élevée, moins vous aurez d'échecs intermittents de refresh de charts dans vos dashboards (à ajuster selon la puissance de votre serveur).
SERVER_WORKER_AMOUNT=64

Effectuez aussi quelques retouches dans docker/pythonpath_dev/superset_config.py pour activer l'alerting et le moteur de template (nécessaire pour créer des datasets avec filtrage dynamique):

FEATURE_FLAGS = {"ALERT_REPORTS": True, "ENABLE_TEMPLATE_PROCESSING": True}

Désactivez la télémétrie en remplaçant dans le fichier docker-compose-non-dev.yml:

x-superset-image: &superset-image apachesuperset.docker.scarf.sh/apache/superset:${TAG:-latest-dev}

par

x-superset-image: &superset-image apache/superset:${TAG:-latest-dev}

Et passez sur la dernière version stable de postgresql en remplaçant:

     image: postgres:14

par

     image: postgres:16

Démarrage

Instanciez et démarrez les conteneurs docker:

docker compose -f docker-compose-non-dev.yml up -d

Superset est donc accessible sur votre serveur de prod via http://127.0.0.1:8088.

Reverse proxy

Configurez un reverse proxy pour sécuriser la connexion, exemple avec caddy:

Editer /etc/caddy/Caddyfile

bi.myawesomecompany.com {
        reverse_proxy http://127.0.0.1:8088
}

Puis redémarrez caddy.

sudo service caddy restart

Première connexion

Connectez vous à https://bi.myawesomecompany.com avec le nom d'utilisateur / mot de passe: admin / admin

Changez votre mot de passe.

Sauvegarde et restauration

Qui dit prod, dit sauvegarde !

Lors de l'édition du fichier de configuration docker-compose-non-dev.yml, vous aurez sans-doute remarqué qu'on instancie une base de données postgresql.

Pour faire la sauvegarde/restauration de superset il n'y a que cette base à prendre en compte.

La sauvegarde peut donc se faire à chaud via un simple:

docker exec -t superset_db pg_dump superset -U superset | xz > backup.sql.xz

Pour la restauration, on ne démarrera au préalable QUE le conteneur postgresql, évitant ainsi d'avoir une instance superset branchée sur une base de données en cours de restauration:

docker compose down
docker compose -f docker-compose-non-dev.yml up db -d
docker exec -t superset_db dropdb -U superset superset
docker exec -t superset_db createdb -U superset superset
xz -dc backup.sql.xz | docker exec -i superset_db psql -U superset -d superset
docker compose -f docker-compose-non-dev.yml up -d

Mettre à jour des milliers de servers linux en un clin d'oeil avec Ctfreak

Jean-Yves Pellé — Mon, 27 Mar 2023 15:04:06 +0000

Vous cherchez à automatiser la mise à jour de vos serveurs sous linux ?

En lançant les mises à jour en parallèle ?

Sans devoir mettre en place une usine à gaz ?

Ne cherchez plus ce tuto est fait pour vous 😉.

Prérequis

1000 serveurs à mettre à jour (par exemple) :
- Ayant pour hostname serverXXXXX.local avec XXXXX allant de 00001 à 01000
- Utilisant une distribution linux basée sur Debian (Debian, Ubuntu, Linux mint, ...)
- Accessibles via SSH sur le port 22 avec la même clef privée et un compte utilisateur adminuser pouvant lancer des commandes sudo sans demande de mot de passe
Une instance de Ctfreak (vous verrez ça s'installe très bien : pas de dépendance, pas de bdd à configurer) avec un compte administrateur, la Free Edition suffira.

Le script

Sur chaque serveur, ce script shell permettra d'effectuer la mise à jour :

#!/bin/bash
export DEBIAN_FRONTEND=noninteractive
sudo apt update 
sudo apt upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold"

On retrouve le classique apt update/upgrade enrichi de quelques éléments permettant de répondre automatiquement à toute question susceptible d'interrompre le script.

NB : Si vos serveurs sont sous Redhat (ou tout autre distribution basé sur RPM), je vous invite à adapter le script en conséquence.

Ctfreak

L'étape suivante sera d'utiliser ctfreak pour déployer et exécuter via SSH le script sur nos 1000 serveurs.

Connectez-vous à ctfreak avec votre compte administrateur.

Ajout de la clef SSH

Allez dans SSH Credential -> Bouton New SSH Credential

Ajoutez la clef privée SSH MySSHKey qui permet de se connecter aux serveurs et validez.

NB : Vous pouvez aussi vous connecter avec un mot de passe plutôt qu'une clef privée

Ajout des nodes

Un node désigne les paramètres de connexion à un serveur via SSH, les nodes sont regroupés en node sources.

Pour rappel, il y a 1000 serveurs, donc 1000 nodes à ajouter.

Il est bien sûr possible d'ajouter les nodes un par un via l'interface web, mais dans ce cas de figure, vous allez plutôt partir sur la création d'un node source dédié qui sera alimenté par un fichier yaml décrivant nos 1000 nodes.

Soit le fichier /home/adminuser/ctfreak-nodes.yaml (accessible en lecture par le user faisant tourner l'instance de ctfreak) :

- name: server0001
  tagNames:
    - debian_server
    - linux_server
  username: adminuser
  hostname: server0001.local
  osFamily: UNIX
  sshPort: 22

- name: server0002
  tagNames:
    - debian_server
    - linux_server
  username: adminuser
  hostname: server0002.local
  osFamily: UNIX
  sshPort: 22

## ...  
## Complétez avec les nodes 3 à 1000
## ...

Allez dans Nodes -> Bouton New External Node Source

Vérifiez que la clef SSH MySSHKey est bien sélectionnée comme Credential (ctfreak s'en servira pour se connecter aux nodes via SSH),
puis validez pour créer le node source Linux Servers (ce dernier resynchronisera ses nodes à partir du fichier yaml toutes les 2 heures).

Si le fichier yaml a bien été reconnu, vous devriez voir la liste suivante :

Ajout de la tâche

Allez dans Projects -> Bouton New Project

Validez pour créer le projet Sysadmin.

Cliquez sur le bouton New Task.

Validez pour créer la tâche Upgrade debian servers.

Cette tâche déploiera et exécutera le script shell :

sur tous les nodes correspondants au tag #debian_server
à chaque 1er jour du mois à 4h du matin

Exécution de la tâche

Pour éviter d'attendre le 1er du mois pour voir ce que ça donne 😉, exécutez la tâche en cliquant sur le bouton Execute.

L'exécution est lancée, cliquez alors sur l'identifiant d'exécution #T0ZJX pour consulter son avancement pour chaque node.

Une fois l'exécution terminée, il se peut que quelques nodes soient en échecs (timeout sur l'ouverture de connexion SSH, repo des packages indispo au lancement du apt-update, problème de DNS, ...) :

Après avoir coché le filtre Failed only pour n'afficher que les nodes en échecs et consulter les logs pour fixer le souci (ici un problème de DNS),
vous pouvez lancer une nouvelle exécution pour retraiter uniquement les nodes en échecs via un simple clic sur Re-execute failed nodes.

Conclusion

Vous voilà avec un process fiable de mise à jour de vos serveurs.

Pour ceux qui voudraient aller plus loin, voici quelques suggestions d'amélioration :

Ajouter une notification par mail en cas d'échec de mise à jour.
Ajouter des serveurs Redhat:
- Ajouter 1000 nouveaux nodes avec les tags redhat_server, linux_server.
- Dupliquer la tâche Upgrade debian servers en Upgrade redhat servers et l'adapter en conséquence.
- Créer une tâche Upgrade all servers de type workflow pour lancer en parallèle (PRO Edition) ou séquentiellement (FREE Edition) les tâches Upgrade debian servers et Upgrade redhat servers.

Mise en place d'un pipeline CI/CD self-hosted

Jean-Yves Pellé — Mon, 27 Mar 2023 14:32:30 +0000

Pour rappel, un pipeline CI/CD (ou pipeline d'intégration et de déploiement continus) est une série d'actions à effectuer en vue de distribuer de manière cohérente et fiable une nouvelle version d'un logiciel.

Dans cet article, nous allons voir comment mettre en place un pipeline CI/CD minimaliste (lancer automatiquement les builds et déploiements lors des événements push sur un dépôt Git) et auto-hébergé que vous pourrez par la suite adapter selon vos besoins.

Prérequis

Il nous faut:

Un serveur de build que nous nommerons buildsrv:
- Sous linux
- Accessible via SSH
- Disposant des outils make et git
Une forge git hébergeant les sources de notre application. Nous utiliserons ici GitHub mais toute autre forge capable d'appeler des webhooks lors de la réception d'évènements push sur ses dépôts fera aussi très bien l'affaire.
Une instance de Ctfreak avec un compte administrateur (la Free Edition suffira). Nous utiliserons ici https://demo.ctfreak.com

Les sources

Les sources de notre application peuvent se résumer à ce Makefile:

build:
    @echo "Build my awesome app"
deploy: build
    @echo "Deploy my awesome app"

Vous pouvez les retrouver dans ce dépôt git, avec la branche master correspondant à la dernière version déployée et la branche dev à la version en cours de développement.

Objectifs

Passons en revue les objectifs à atteindre par la mise en place de notre pipeline CI/CD:

Pour la partie Intégration Continue:
- Compiler les sources (via la commande make) à chaque fois qu'un développeur push sur la branche dev.
- Recevoir une notification en cas d'échec.
Pour la partie Déploiement Continue:
- Compiler les sources et déployer (via la commande make deploy) à chaque fois qu'un développeur push sur la branche master.
- Recevoir une notification en cas de succès ou d'échec.

Pour ce faire Ctfreak va jouer le rôle de passerelle entre la forge git et le serveur de build.

C'est à dire qu'à chaque push reçu par la forge:

Celle-ci va appeler un webhook défini dans Ctfreak.
Ctfreak va alors se connecter via SSH au serveur de build pour exécuter la commande make adéquate, puis envoyer une notification.

Configuration du serveur de build

Connectez-vous à votre serveur de build et lancez les commandes suivantes:

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.master
> Cloning into 'myawesomeapp.master'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 3.59 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.dev
> Cloning into 'myawesomeapp.dev'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 7.19 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ cd myawesomeapp.dev

ctfreakdemo@buildsrv:~/myawesomeapp.dev$ git checkout dev
> Branch 'dev' set up to track remote branch 'dev' from 'origin'.
> Switched to a new branch 'dev'

Comme vous pouvez le constater, nous avons cloné le repo git 2 fois (1 par branche), ce qui nous permettra d'éviter d'éventuels conflits si les builds des branches master et dev devaient s'exécuter de manière concurrente.

Configuration de Ctfreak

Connectez vous à votre instance Ctfreak avec un compte administrateur.

Ajout du serveur de build

Commencez par ajouter la clef SSH permettant de se connecter au serveur de build via SSH Credential → New SSH Credential:

Puis ajoutez le serveur en lui même via Nodes → Internal Nodes → New node:

Vérifiez que la clef SSH Key for buildsrv est bien sélectionnée comme Credential (Ctfreak s’en servira pour se connecter au serveur), puis validez pour créer le node buildsrv.

Création du projet CI/CD

Nous allons créer un projet dédié pour rassembler toutes les tâches en rapport avec notre pipeline CI/CD.

Allez dans Projects → New Project

Validez pour créer le projet.

Ajout d'un notifieur

Allez dans Projects → CI/CD → Notifiers → New notifier

Choisissez Discord comme type de notifieur (par exemple).

Précisez l'URL du webhook discord à appeler et validez.

Une notification de test peut être envoyée en cliquant sur le bouton d'envoi (celui en forme d'avion en papier).

Création de la tâche de build

Allez dans Projects → CI/CD → New task

Choisissez Bash Script comme type de tâche.

Comme vous l'aurez deviné, cette tâche va permettre de compiler la branche dev sur le serveur de build et envoyer une notification en cas d'échec.

La stratégie d'exécution multiple (multiple execution policy): chaînage intelligent (smart chaining) choisie ici va permettre d'éviter de lancer des builds superflus.

Prenons un exemple: vous avez un build qui dure une demi-heure et 3 pushs sont envoyés sur le dépôt git à 5 minutes d'intervalle.

Au premier push, le build est lancé immédiatement.

Au 2ème push, un 2ème build est mis en attente vu que le premier n'est pas encore terminé.

Au 3ème push, on ne prévoit pas de faire de 3ème build car le 2ème est toujours en attente.

Création de la tâche de déploiement

Pour gagner du temps, créez cette tâche à partir de celle de build via Projects → CI/CD → Buid myawesomeapp / dev → Duplicate.

En y apportant les modifications suivantes:

Nous souhaitons recevoir une notification en fin de traitement dans tous les cas.
Le contenu du script est mis à jour pour déployer à partir de la branche master.

Ajout des webhooks

Maintenant que nos tâches de build et de déploiement sont créées, nous allons ajouter pour chacune d'elle un webhook entrant qui pourra être appelé par notre forge git.

Commençons avec la tâche de déploiement en allant dans Projects → CI/CD → Buid myawesomeapp / dev → Incoming webhooks → New webhook.

Choisissez GitHub comme type de webhook.

Lorsque GitHub appelera notre webhook suite à un push, il le fera quelque soit la branche concernée or notre tâche ne doit être exécuté que lorsque ce push concerne la branche dev.

Pour résoudre ce problème, nous ajoutons une condition d'exécution qui s'appliquera au payload JSON envoyé par GitHub. La tâche ne sera exécutée que si la condition est remplie (c-a-d le payload fait bien référence à la branche dev).

Définissez au besoin un secret qui permettra à Ctfreak de valider que les appels au webhook proviennent bien de GitHub et validez.

Une fois le webhook créé, nous récupérons son URL pour le paramétrage de GitHub.

Au niveau de la configuration de votre dépôt GitHub, ajoutez le webhook que nous venons de créer en précisant bien:

Son URL et son secret
Que c'est le format JSON qui est attendu
Que le webhook ne doit être appelé QUE pour les évènements push

Procédez de la même façon (création du webhook, référencement dans GitHub) pour la tâche de déploiement (pensez à bien faire référence à la branche master dans la condition d'exécution):

Test du pipeline

Maintenant que tout est en place, voyons ce que provoque un git push sur la branche dev:

Suite à l'appel du webhook GitPushDev la tâche de build a bien été exécutée.

En cliquant sur l'icône en forme d'œil, nous avons les logs de l'exécution.

Même test avec un git push sur la branche master:

Ici c'est le webhook GitPushMaster qui a été appelé comme prévu.

Et le petit plus ici, c'est la réception d'une notification Discord pour confirmer que le déploiement s'est bien passé.

Conclusion

Vous voilà avec un pipeline CI/CD opérationnel et suffisamment souple pour prendre en charge les workflows de build les plus complexes.

En optant pour cette solution self-hosted et en plaçant le cœur du processus de build dans un script shell plutôt que dans un logiciel d'intégration continue comme Jenkins, GitHub Actions, ..., vous bénéficierez de plusieurs avantages:

Vous ne serez pas contraint par les limites de ce dernier (version de git pas la plus récente, non prise en charge de certaines spécificités de votre build, limite du nombre de build journalier souscrit atteinte, ...).
Si votre serveur de build est indisponible, vous pourrez encore effectuer un déploiement en urgence à partir d'un poste de développeur (croyez-moi, ça peut servir).

Setting up a self-hosted CI/CD pipeline

Jean-Yves Pellé — Wed, 22 Mar 2023 23:27:49 +0000

As a reminder, a CI/CD pipeline (or Continuous Integration/Continuous Deployment pipeline) is a series of actions to be performed in order to consistently and reliably distribute a new version of a software.

In this article, we will see how to set up a minimalistic CI/CD pipeline (automatically triggering builds and deployments upon push events to a git repository) that is self-hosted and can be adapted according to your needs.

Prerequisites

We will need:

A build server that we will name buildsrv:
- Running Linux
- Accessible via SSH
- With make and git tools
A git forge hosting the sources of our application. We will use GitHub here, but any other forge capable of calling webhooks upon receiving push events on its repositories will also do the job.
An instance of Ctfreak with an administrator account (the Free Edition will suffice). We will use https://demo.ctfreak.com here.

The sources

The sources of our application can be summarized in this Makefile:

build:
    @echo "Build my awesome app"
deploy: build
    @echo "Deploy my awesome app"

You can find them in this git repository, with the master branch corresponding to the latest deployed version and the dev branch to the version currently under development.

Objectives

Let’s review the objectives to be achieved by setting up our CI/CD pipeline:

For the Continuous Integration part:
- Compile the sources (via the make command) every time a developer pushes to the dev branch.
- Receive a notification in case of failure.
For the Continuous Deployment part:
- Compile the sources and deploy (via the make deploy command) every time a developer pushes to the master branch.
- Receive a notification in case of success or failure.

To do this, Ctfreak will act as a gateway between the git forge and the build server.

That is, every time a push is received by the forge:

It will call a webhook defined in Ctfreak.
Ctfreak will then connect via SSH to the build server to execute the appropriate make command, then send a notification.

Build server configuration

Connect to your build server and run the following commands:

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.master
> Cloning into 'myawesomeapp.master'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 3.59 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ git clone https://github.com/jypsoftware/myawesomeapp.git myawesomeapp.dev
> Cloning into 'myawesomeapp.dev'...
> remote: Enumerating objects: 15, done.
> remote: Counting objects: 100% (15/15), done.
> remote: Compressing objects: 100% (13/13), done.
> remote: Total 15 (delta 2), reused 6 (delta 1), pack-reused 0
> Receiving objects: 100% (15/15), 14.38 KiB | 7.19 MiB/s, done.
> Resolving deltas: 100% (2/2), done.

ctfreakdemo@buildsrv:~$ cd myawesomeapp.dev

ctfreakdemo@buildsrv:~/myawesomeapp.dev$ git checkout dev
> Branch 'dev' set up to track remote branch 'dev' from 'origin'.
> Switched to a new branch 'dev'

As you can see, we have cloned the git repo 2 times (1 per branch), which will allow us to avoid potential conflicts if the builds of the master and dev branches were to run concurrently.

Ctfreak Configuration

Adding the build server

Start by adding the SSH key to connect to the build server via SSH Credential → New SSH Credential:

Then add the server itself via Nodes → Internal Nodes → New node:

Make sure that the SSH Key for buildsrv key is selected as the Credential (Ctfreak will use it to connect to the server), then validate to create the buildsrv node.

Creating the CI/CD project

We are going to create a dedicated project to gather all the tasks related to our CI/CD pipeline.

Go to Projects → New Project

Validate to create the project.

Adding a notifier

Go to Projects → CI/CD → Notifiers → New notifier

Choose Discord as the notifier type (for example).

Specify the discord webhook URL to call and validate.

A test notification can be sent by clicking on the send button (the one shaped like a paper airplane).

Creating the build task

Go to Projects → CI/CD → New task

Choose Bash Script as the task type.

As you may have guessed, this task will compile the dev branch on the build server and send a notification in case of failure.

The smart chaining multiple execution policy selected here will avoid launching unnecessary builds.

Let’s take an example: you have a build that lasts half an hour and 3 pushes are sent to the git repository at 5-minute intervals.

At the first push, the build is launched immediately.

At the 2nd push, a 2nd build is put on hold since the first one is not yet finished.

At the 3rd push, we don’t plan to do a 3rd build because the 2nd one is still on hold.

Creating the deployment task

To save time, create this task from the build task via Projects → CI/CD → Buid myawesomeapp / dev → Duplicate.

Making the following modifications:

We want to receive a notification in case of success or failure.
The script content is updated to deploy from the master branch.

Adding webhooks

Now that our build and deployment tasks are created, we will add an incoming webhook for each one that can be called by our git forge.

Let’s start with the deployment task by going to Projects → CI/CD → Buid myawesomeapp / dev → Incoming webhooks → New webhook.

Choose GitHub as the webhook type.

When GitHub calls our webhook following a push, it will do so regardless of the branch involved, but our task should only be executed when this push concerns the dev branch.

To solve this issue, we add an execution condition that will apply to the JSON payload sent by GitHub. The task will only be executed if the condition is met (i.e. the payload does indeed refer to the dev branch).

Define a secret if necessary, which will allow Ctfreak to validate that calls to the webhook come from GitHub, and validate.

Once the webhook is created, we retrieve its URL for GitHub configuration.

In your GitHub repository settings, add the webhook we just created, specifying:

Its URL and secret
That the expected format is JSON
That the webhook should be called ONLY for push events

Proceed in the same way (create the webhook, reference it in GitHub) for the deployment task (be sure to reference the master branch in the execution condition):

Pipeline testing

Now that everything is set up, let’s see what happens when we do a git push on the dev branch:

After calling the GitPushDev webhook, the build task was executed successfully.

By clicking on the eye-shaped icon, we can see the execution logs.

Let’s do the same test with a git push on the master branch:

This time, the GitPushMaster webhook was called as expected.

And the added bonus here is the receipt of a Discord notification confirming that the deployment was successful.

Conclusion

You now have an operational CI/CD pipeline that is flexible enough to handle even the most complex build workflows.

By opting for this self-hosted solution and placing the core of the build process in a shell script rather than in a continuous integration software like Jenkins, GitHub Actions, etc., you will benefit from several advantages:

You will not be constrained by the limitations of the latter (older version of Git, certain specificities of your build not supported, limit of the number of daily builds reached, etc.).
If your build server is unavailable, you can still perform an emergency deployment from a developer’s workstation (trust me, it can be useful).

Introduction to web scraping with goquery

Jean-Yves Pellé — Sat, 17 Apr 2021 16:27:40 +0000

The use of an API is the preferred solution to retrieve data, but when it does not exist, is incomplete, or is subject to limitations that are too restrictive, scraping the associated website can be a good alternative.

What we call web scraping is the collection (via the writing of bots called scrapers) of targeted data present on a set of web pages. A scraper is designed specifically for a given site, unlike a crawler which will be more generalist (Google Shopping uses scrapers when Google (aka the search engine) uses crawlers).

The uses are multiple (aggregation of ads from several job boards, competitive intelligence, …), and are not necessarily reserved to the professional world.

We can indeed find several situations where writing a personal scraper can have its interest:

Sending an email as soon as the homework has been updated on the children’s school blog
Automatically setting the alarm clock 15 minutes earlier when a weather site forecasts ice the next morning
Receive a notification when the price of an item on an e-commerce site falls below a certain threshold

Let’s illustrate this last use case.

Looking for a cheap Raspberry Pi 4

Let’s take this Raspberry Pi 4B on www.kubii.fr, it is on sale for 83.99€ but we would like to wait until it drops below 75€ before ordering. As a price drop can quickly lead to a stockout, we might as well be warned quickly to avoid missing a good deal.

The information we are looking for here is the price of the item:

Let’s see how it looks in the HTML:

<p class="our_price_display" itemprop="offers" itemscope="" itemtype="https://schema.org/Offer">
  <link itemprop="availability" href="https://schema.org/InStock">
  <span id="our_price_display" class="price" itemprop="price" content="83.99">83,99 €</span> TTC
  <meta itemprop="priceCurrency" content="EUR">
</p>

From there, we need to find an anchor point, that is to say a criterion that will allow us to find the price (i.e. the attribute content of the span tag) for sure and without duplication.

After analyzing the complete page, it turns out that the itemprop="price" attribute is a good candidate (we only find one span tag meeting this criterion).

Our scraper (in go) can now take shape:

package main

import (
    "github.com/PuerkitoBio/goquery"
    "log"
    "net/http"
    "strconv"
    "time"
)

func main() {
    goodPrice := false

    // Stay in the loop as long as we have not passed below our threshold

    for !goodPrice {

        // Loading of the product page
        res, err := http.Get("https://www.kubii.fr/cartes-raspberry-pi/2955-raspberry-pi-4-modele-b-8gb-0765756931199.html")
        if err != nil {
            log.Printf("Unable to load the product page, we will try again later: %v", err)
            time.Sleep(30 * time.Minute)
            continue
        }
        if res.StatusCode != 200 {
            res.Body.Close()
            log.Printf("Unable to load the product page, we will try again later, status code: %d", res.StatusCode)
            time.Sleep(3 * time.Hour)
            continue
        }

        // HTML Parsing through goquery
        doc, err := goquery.NewDocumentFromReader(res.Body)
        res.Body.Close()
        if err != nil {
            log.Panicf("Unable to parse HTML content: %v", err)
        }

        // Price retrieval
        var priceStr string
        var ok bool

        if priceStr, ok = doc.Find("span[itemprop='price']").Attr("content"); !ok {
            log.Panicf("Price not found, scraper should be updated")
        }

        // Price conversion
        price, err := strconv.ParseFloat(priceStr, 64)
        if err != nil {
            log.Panicf("Unable to read the price, scraper should be updated")
        }

        if price >= 75 {
            log.Printf("%0.2f € it's still too expensive!", price)
        } else {
            log.Printf("Only %0.2f €, go go go!!!", price)
            // TODO: Sending an email, an SMS, calling a discord webhook, triggering the alarm, flashing the connected bulbs, etc ...
            goodPrice = true
        }

        // We try again in 30 minutes
        time.Sleep(30 * time.Minute)

    }

    log.Printf("Done!")

}

The logic is quite simple

Every 30 minutes, we retrieve the price until it falls below 75€, if necessary we raise an alert.
A basic (and perfectible!) management of the unavailability of the product page is applied, whether they are
- due to us (network failure, … ) => new attempt after 30 minutes
- or due to the site (maintenance, anti-bot defense, …) => new attempt after 3 hours (no need to be stubborn)
If the price is no longer retrievable, we assume that the site must have been updated and that our anchor point must have jumped at the same time, so we must find a new one and update the scraper accordingly.

Note that the search for the famous <span> tag containing our price is easily done via the excellent goquery library, which allows, as in javascript, to use CSS selectors to locate DOM elements.

Finally

A small VPS, a NAS, an old Raspberry PI (hence the purchase of a new one!) or simply your PC (if you’re the kind of person who keeps it on 24 hours a day), and you’re ready to deploy your first scraper.

Feel free to adapt this example to your needs but, word to the wise, be sure to stay reasonable on the frequency of requests, some sites do not like bots that monopolize too many resources on their server to the detriment of real visitors. So you can understand them if they end up blacklisting your IP because you try to retrieve a price every 2 seconds instead of 30 minutes…