DEV Community: Jérôme Dx

Mise : The ultimate Dev Tool Manager for seamless workflows

Jérôme Dx — Mon, 26 Jan 2026 09:00:00 +0000

Tired of juggling multiple versions of your favorite tools, or just want to stay up to date with the latest CLI tools, programming languages, and more ?

Mise (mise-en-place) is a utility written in Rust, that allows you to manage the versions of your developer tools on different projects. It’s very fast and very complete, with different ways to use it :

Dev Tools : A version manager for developer tools (alternative to ASDF, Pyenv, Tfenv/Tenv, etc.)
Environments : Allows to load your environment variables, by projects (alternative to Direnv)
Tasks : Allows to run defined tasks for your projects (alternative to Make)

Let’s explore how it can streamline your daily workflow.

Getting Started with Mise

Installation

Mise is available on all platforms that support Rust, i.e. Linux, Windows or MacOS.

For shell users, start by installing Mise with a single command :

curl https://mise.run | sh

Then, add the following line to your .zshrc to enable Mise in your shell :

echo "eval $(/home/user/.local/bin/mise activate zsh)" >> "/home/user/.zshrc"
source ~/.zshrc

Dev Tools

As a version manager, Mise handle the different version of your tools for development. It can be used for local development and inside your CICD pipelines.

Basic Commands

Check your Mise version :

mise version

Get help :

mise --help

Installing Tools

Install the latest versions of your favorite tools effortlessly :

mise use python@latest
mise use go@latest
mise use terraform@latest

Need cloud tools? Mise has you covered :

mise use gcloud@latest
mise use scaleway@latest

Managing Your Tools

List installed software :

mise ls

Explore available versions :

mise ls-remote terraform
mise ls-remote terraform@1.11

Search for tools :

mise search jq

Staying Up to Date

Update your tools interactively :

mise up --interactive

You can also update Mise itself :

mise self-update

Environments

Mise allows you to manage environment variables per project, ensuring your tools and scripts always run in the correct context. This is especially useful for projects requiring specific configurations or secrets.

Set environment variables for a project

Create a .mise.toml file in your project root and define your environment variables :

[env]
DATABASE_HOST ="localhost:5432"
API_KEY ="your_api_key_here"

Mise automatically loads these variables when you enter the project directory, thanks to its shell integration, you just need to trust the file.

mise trust

Handle multiple environments

File .mise.dev.toml :

[env]
DATABASE_HOST ="dev-db:5432"
API_KEY ="dev_key"

File .mise.prd.toml :

[env]
DATABASE_HOST ="prd-db:5432"
API_KEY ="prd_key"

You can switch between environments like this :

MISE_ENV=dev mise env | grep API_KEY
MISE_ENV=prd mise env | grep API_KEY

Tasks

Mise simplifies running repetitive or complex tasks by allowing you to define and execute them directly from your project configuration.

Define tasks

Add a [tasks] section to your .mise.toml file :

[tasks]
start ="docker-compose up"
test ="pytest"
build ="cargo build --release"
deploy ="make deploy"

Run a task

Execute any defined task with :

mise run <task_name>

# Examples:
mise run test
mise run deploy

List available tasks

To see all tasks defined in your project :

mise tasks

Why Mise?

Simple : Clear and intuitive syntax.
Versatile : Replaces many tools.
Lightweight : Written in Rust, fast and resource-efficient.
Efficient : Effortlessly manages versions and dependencies.

Here is a Mise cheatsheet you can consult to remember some common commands.

There are still a lot of possibilities you can explore with this tool, like securing your variables, execute hooks or even use plugins.

Now you’re all set ! Try it out and see how Mise simplifies your workflow.

GCP-Nuke to clean your sandbox projects

Jérôme Dx — Mon, 25 Aug 2025 17:00:00 +0000

GCP Nuke is a tool you can use to clean up your GCP Projects. It’s a Go client, under MIT licence, that will remove all the resources from your project, or you can restrict them with filters.

As “nuke” let you guess, it can be a very destructive tool.

Some equivalents exist for AWS and Azure, provided by the same author.

It is particularly interesting, by example, if you have a Sandbox project you want to clean regularly, for cost reasons or even sustainibility considerations.

I will show you concretly how you can use it, for this kind of use.

A basic Sandbox usage

FIrst you can install it simply with brew :

brew install ekristen/tap/gcp-nuke

Ensure you have all this services enabled :

gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable compute.googleapis.com
gcloud services enable storage.googleapis.com
gcloud services enable run.googleapis.com
gcloud services enable cloudfunctions.googleapis.com
gcloud services enable pubsub.googleapis.com
gcloud services enable iam.googleapis.com

# to check everything is correct :
gcloud services list --enabled

You will have the possibility to customize the behavior of the tool with a config file.

Here is the list of available resources handle by the tool.

For my usage, I choosed to select some resources :

Cloud Functions
CloudRun
GKE
Compute Instances & disks
Storage Buckets
BigQuery Datasets
CloudSQL instances

It will parse the 3 regions I use most, and it let me choose to keep some resources with this labels :

managed_by: terraform
gcp-nuke: ignore

regions:
  - global
  - us-east1 # US: South Carolina
  - europe-west1 # EU: Belgium
  - europe-west9 # EU: Paris

accounts:
  my-gcp-project:
      includes:
        - CloudFunction
        - CloudFunction2
        - CloudRun
        - GKECluster
        - ComputeInstance
        - ComputeFirewall
        - ComputeForwardingRule
        - ComputeSSLCertificate
        - ComputeDisk
        - StorageBucket
        - BigQueryDataset
        - CloudSQLInstance

    filters:
      __global__:
        - property: label:gcp-nuke
          value: "ignore"
        - property: label:managed_by
          value: "terraform"
      ComputeFirewall:
        - property: "name"
          value: "default-*"
          invert: true

To authenticate, you can create a Service Account and declare the environment variable, or connect with the "auth login" option :

export GOOGLE_APPLICATION_CREDENTIALS=creds.json
# or
gcloud auth login

You can do a “dry-run” launch to see what will be destroyed (dry-run is defined by default to limit the risks) :

gcp-nuke run --config config.yml --no-prompt --project-id <current-gcp-project-id>

When you’re sure, you can launch the “nuke” 💥 :

gcp-nuke run --config config.yml --no-prompt --no-dry-run --project-id <current-gcp-project-id>

Automation

If you want, you can define a regular job that will clear up the resources, once a day at 2 am, for instance.

This can be achieved with a CLoudRun job.

Here is the Dockerfile :

FROM docker.io/library/golang:1.23 AS builder

# Install gcp-nuke
RUN go install github.com/ekristen/gcp-nuke@latest

FROM docker.io/library/debian:bookworm-slim

# Install ca-certificates for HTTPS requests and gcloud CLI
RUN apt-get update && \
    apt-get install -y ca-certificates curl gnupg && \
    echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && \
    apt-get update && \
    apt-get install -y google-cloud-cli && \
    rm -rf /var/lib/apt/lists/*

COPY --from=builder /go/bin/gcp-nuke /usr/local/bin/gcp-nuke

WORKDIR /app
COPY config.yml .
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh

CMD ["./entrypoint.sh"]

And the entrypoint.sh :

#!/bin/bash
set -e

echo "Setting project..."
gcloud config set project ${PROJECT_ID}

echo "Running gcp-nuke..."
gcp-nuke run --config config.yml --no-prompt --no-dry-run --project-id ${PROJECT_ID}

echo "gcp-nuke completed."

Push it on the Artifact Registry

gcloud artifacts repositories create gcp-nuke-repo \
  --repository-format=docker \
  --location=europe-west9 \
  --description="Docker Repo for gcp-nuke"

podman build -t europe-west9-docker.pkg.dev/<your-project-id>/gcp-nuke-repo/gcp-nuke-job:latest .

gcloud auth configure-docker europe-west9-docker.pkg.dev
podman push europe-west9-docker.pkg.dev/<your-project-id>/gcp-nuke-repo/gcp-nuke-job:latest

Here is the Terraform for deployement, that will create the CloudRun Job, the Service Account and the Cloud Scheduler :

# ----------------------------
# Cloud Run Job
# ----------------------------
resource "google_cloud_run_v2_job" "gcp_nuke_job" {
  name     = "gcp-nuke-job"
  location = var.region

  deletion_protection = false

  template {
    template {
      service_account = google_service_account.gcp_nuke.email

      containers {
        image = "${var.region}-docker.pkg.dev/${var.project_id}/gcp-nuke-repo/gcp-nuke-job:latest"

        env {
          name  = "PROJECT_ID"
          value = var.project_id
        }

        resources {
          limits = {
            cpu    = "1"
            memory = "512Mi"
          }
        }
      }

      max_retries = 3
      timeout     = "300s"
    }

    parallelism = 1

    labels = {
      managed_by = "terraform"
    }
  }
}

# ----------------------------
# Service account
# ----------------------------

resource "google_service_account" "gcp_nuke" {
  account_id   = "sa-gcp-nuke"
  display_name = "Service Account for gcp-nuke"
}

resource "google_project_iam_member" "gcp_nuke_owner" {
  project = var.project_id
  role    = "roles/owner"
  member  = "serviceAccount:${google_service_account.gcp_nuke.email}"
}

# ----------------------------
# Cloud Scheduler
# ----------------------------
resource "google_cloud_scheduler_job" "gcp_nuke_schedule" {
  name        = "gcp-nuke-schedule"
  description = "Run gcp-nuke every day at 2am"
  schedule    = "0 2 * * *"
  time_zone   = "Europe/Paris"

  http_target {
    http_method = "POST"
    uri         = "https://${var.region}-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/${var.project_id}/jobs/${google_cloud_run_v2_job.gcp_nuke_job.name}:run"

    oauth_token {
      service_account_email = google_service_account.gcp_nuke.email
    }
  }
}

GCP-Nuke is a convenient tool to clean up a GCP Project, you have now some keys to use it locally, or to automate it.

If you want to regularly clean up a Sandbox project, this is a case where it's particularly suitable, as it reduces financial costs and the ecological footprint, always with the same idea to build only what you really consume.

Cloud Run: Container management made simple

Jérôme Dx — Wed, 02 Jul 2025 22:00:00 +0000

If you are looking for a simpler service than Kubernetes to deploy and manage containers on an environment (even in prod), Google Cloud offers a turnkey service that could be used, which is called Cloud Run.

The purpose of this article is to go a little further than the Quickstart, to show you how to deploy a simple python container, via Terraform.

Prerequisites for the demo

First, you will need this elements :

Terraform: Install Terraform
Google Cloud SDK: Install gcloud
A Google Cloud Platform (GCP) Project: Create a GCP project

Then, create the Cloudbuild trigger in the GCP console :

Go to the Cloud Build Triggers page.
Click on Create Trigger.
Set the following options:
- Name: your-github-trigger
- Event: Push to a branch
- Source: GitHub
- Repository: Select the repository you created for this project.
- Branch: main
- Build Configuration: Cloud Build configuration file (yaml or json)
Click Create to finish setting up the trigger.

Note : create a service account with the name sa-cloudbuild and assign it the Cloud Build Service Account and Storage Admin roles. This service account will be used by Cloud Build to deploy the application to Cloud Run.

Terraform resources

To create a repository in Artifact Registry :

resource "google_artifact_registry_repository" "repo" {
  location      = var.region
  repository_id = "flask-repo"
  description   = "Repository for Flask app"
  format        = "DOCKER"

  labels = {
    managed_by = "terraform"
  }
}

Create the simple app :

# main.py
import os

from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    """Example Hello World route."""
    name = os.environ.get("NAME", "World")
    return f"Hello {name}!"

if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

With this requirements :

# requirements.txt
Flask==3.0.3
gunicorn==23.0.0
Werkzeug==3.0.3

And this Dockerfile :

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt main.py ./

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8080

ENV NAME=World
ENV PORT=8080

CMD ["gunicorn", "--bind", "0.0.0.0:8080", "main:app"]

Here is the terraform code to build & push the container with Cloud Build :

resource "null_resource" "build_image" {
  triggers = {
    dockerfile_hash   = filemd5("${path.root}/app/Dockerfile")
    main_py_hash      = filemd5("${path.root}/app/main.py")
    requirements_hash = filemd5("${path.root}/app/requirements.txt")
  }

  provisioner "local-exec" {
    command = <<-EOT
      gcloud builds submit ${path.root}/app \
        --tag ${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.repo.repository_id}/flask-app:latest \
        --project ${var.project_id}
    EOT
  }

  depends_on = [
    google_artifact_registry_repository.repo
  ]
}

For our example, I choosed to launch it directly from Terraform, without the necessity to have Docker or Podman installed, just Gcloud Cli.

And finally the Terraform code to create the Cloud Run Service :

# Cloud Run Service
##

resource "google_cloud_run_v2_service" "flask_service" {
  name     = var.service_name
  location = var.region

  labels = {
    managed_by = "terraform"
  }

  template {
    labels = {
      managed_by = "terraform"
    }

    containers {
      image = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.repo.repository_id}/flask-app:latest"

      ports {
        container_port = 8080
      }

      env {
        name  = "NAME"
        value = "Cloud Run"
      }

      resources {
        limits = {
          cpu    = "1000m"
          memory = "512Mi"
        }
      }
    }

    scaling {
      min_instance_count = 0
      max_instance_count = 3
    }
  }

  depends_on = [
    null_resource.build_image
  ]
}

# IAM policy to allow public access
##

resource "google_cloud_run_service_iam_member" "public_access" {
  service  = google_cloud_run_v2_service.flask_service.name
  location = google_cloud_run_v2_service.flask_service.location
  role     = "roles/run.invoker"
  member   = "allUsers"
}

This outputs will give you the urls generated for your service, and the registry :

output "service_url" {
  description = "URL of the Cloud Run service"
  value       = google_cloud_run_v2_service.flask_service.uri
}

output "artifact_registry_url" {
  description = "Artifact Registry repository URL"
  value       = google_artifact_registry_repository.repo.name
}

Now you can launch a plan and apply your code, you have a container you can work on, handle a service managed by Google Cloud.

All the code is available in this github repository.

There is an “Awesome CloudRun” page where you will find a lot of interesting usages for Cloud Run.

Google also offers some Codelabs where you can get trained and explore advanced use cases with Cloud Run.

Now you can use and improve your configuration to work with your containers, have fun with it.

Python: The Nim game

Jérôme Dx — Mon, 17 Mar 2025 14:00:00 +0000

Today we are going to look at a small algorithm exercise in python, through the game called the nim game.

The rule

This is a two-player game. We have n matches to start with (for example 10).
In turn, each player will take 1, 2 or 3 matches. The player who takes the last match loses.

The algorithm

This is a fairly deterministic game, in the sense that when you know the trick, you can't lose (unless you started from a losing position).

Here is a table of the right number of matches to take (x representing a losing position):

We can therefore observe an infinite periodic loop, with a period of 4.

The player has lost (if the other plays well), when we have a modulo 4 = 1 (in other words the remainder of the division is equal to 1).

The winning strategy is therefore to send the other player to a losing position.

1-player version

We will formalize this in the algorithm, using the modulo function, to write the computer's AI.

n = int(input("Enter the initial number of matches: "))
print("Number of matches : ", n)
while n>0:
    j=-1
    print("P1 turn")
    while j<1 or j>3 or j>n:
        j=int(input("How much do you take ? (1-3) "))
    n=n-j
    if n==0:
        print("You lost !")
    else:
        print("P2 turn (cpu)")
        print("Remaining ", n)
        if n%4==3:
            p=2
        elif n%4==2:
            p=1
        elif n%4==0:
            p=3
        else:
            p=1
        print("I take some", p)
        n=n-p
        print("Remaining ", n)
        if n==0:
            print("I lost !")

Two-players version

Here is a version if you want experiment with a friend, keeping in mind this strategy :

def nim_game():
    print("Welcome to the Nim Game!")
    print("Rules: Players take turns removing 1 to 3 objects from the pile.")
    print("The player forced to take the last object loses.")

    # Initialize the pile
    pile = int(input("Enter the initial number of objects in the pile: "))
    while pile <= 0:
        print("The pile must contain at least 1 object.")
        pile = int(input("Enter the initial number of objects in the pile: "))

    # Player names
    player1 = input("Enter Player 1's name: ")
    player2 = input("Enter Player 2's name: ")

    # Start the game
    current_player = player1

    while pile > 0:
        print(f"\nThere are {pile} objects in the pile.")
        print(f"{current_player}'s turn.")

        # Get the number of objects to remove
        try:
            remove = int(input("How many objects will you take (1-3)? "))
            if remove < 1 or remove > 3:
                print("Invalid move. You can only take 1, 2, or 3 objects.")
                continue
            if remove > pile:
                print("Invalid move. You cannot take more objects than are in the pile.")
                continue
        except ValueError:
            print("Invalid input. Please enter a number between 1 and 3.")
            continue

        # Update the pile
        pile -= remove

        # Check if the game is over
        if pile == 0:
            print(f"\n{current_player} is forced to take the last object and loses!")
            break

        # Switch players
        current_player = player2 if current_player == player1 else player1

    print("Game over. Thanks for playing!")

# Run the game
if __name__ == "__main__":
    nim_game()

Here is a little logic exercise, that you can use to practice algorithms in Python.

Also, you know the secret and will not be fooled by this game anymore, have fun!

Create spot instances on GCP & AWS

Jérôme Dx — Mon, 13 Jan 2025 18:00:00 +0000

When you create virtual machines on Cloud providers, the price depends on the size on the machine and the region where you create it.

You have the possibility to reserve capacity for a long period of time (1 year for instance), to have reductions from the provider.

Another way to have reductions is to get Spot instances : it offers reduced price machines based on spare resources not used by other machines.

The advantage is that you have very interesting price for you virtual machines, the downside is that you should consider this resources can break at any moment.

For developement and testing environments, that’s a very good option.

You have to handle the automated rebuild of the instances to work properly. That’s why we will build a Managed Instance Group on GCP and an Autoscaling Group on AWS.

We are going to see how to implement that on both providers.

On Google Cloud

First, you need to create an Instance Template, in the Compute Engine section :

Give a name to this Instance Template, and select a region (we choose one where there are free credits) :

We choose e2-micro instance (General purpose with 1 Gb RAM), we can see that the cost is very low for one month, including storage : $3.44 :

You can set some firewall rules :

And assign to your prefered VPC, then create :

Now you will create a MIG (Managed Instance Group), to create the VMs according to this template :

Create a stateless Instance Group, give a name and select the template :

Set a location with multiple zones, set an Autoscaling to have permanently 2 machines running, then create the MIG :

Now you can access your VMs, by clicking “Connect SSH” :

On AWS

First, you need to create a Launch Template, in the EC2 section :

Give a name to this Launch Template :

Select an AMI from the catalog, Amazon Linux will allow to easily connect with session manager, and Arm will use less resources and cheaper :

Select t4g.nano, that is an optimized general purpose instance, with a base price of 0.0046 USD per hour :

Select the VPC and Security group you want to use (for the SG, just allow All traffic in Outbound, for now) :

Specify a tag “Name” for your instances to recognize them in the list later :

In Advanced details, specify an IAM role for SSM acces, and that you want Spot instances, then create the template :

Now, create an Auto Scaling Group (ASG) :

Name it and select the template just created :

Select the network options you prefer :

Select the scaling policies you want, then you can create the ASG :

Now you can access your VMs, using Session Manager :

Conclusion

You can see that this mechanism is really similar on both of these cloud providers and Spot instances are really easy to implement.

Of course, the best option will be to automate it an IaC tool like Terraform or OpenTofu and to optimize it according to your needs.

You now have the keys to implement them on your projects, to have low-cost computing resources for any development objective.

Set budget alerts on GCP & AWS

Jérôme Dx — Mon, 16 Dec 2024 15:28:19 +0000

To manage costs on your cloud infrastructure, you can use an external tool (like Tailwarden for instance), but you can also work with native tools from your cloud provider.

This is the goal of this article, we will see how to do it simply on Google Cloud and AWS.

The main advantages are that you will be very close to the vendor's philosophy and you will limit the number of tools you use.

On Google Cloud

The goal here is simply to set a budget limit and you will be alerted if it is exceeded.

First, go to the “Budget & alerts” section and clic on “Create a budget” :

Note : You should select the Billing account associated to the project you want to manage.

Then enter a name for your alert and delimit according to your needs, by projects and services (in our case we will make a global alert) :

Select an amount to be alerted :

You can configure some thresholds and define who receives the alerts (the users can be setup in the Billing account management section) :

Now your alert is ready and you will be notified if necessary :

On AWS

First, go to the “Billing and Cost Management” section and clic on “Create budget” :

Then choose the template “Monthly cost budget” and enter a name for your alert :

Select an amount to be alerted and email recipients, by default this template applies to all AWS services in the current account, and thresolds are defined (you can customize them later) :

Now your alert is ready and you will be notified if necessary :

Conclusion

You now have the keys to manage your budget alerts simply on these two cloud providers.

The advantage with native tools is that you have reduced costs and you directly adhere to the philosophy of the cloud provider. In return, you will not have a centralized interface for a multi-cloud context.

It's now up to you to decide what is most interesting in your case.

Bootstrap your projects with Docker init

Jérôme Dx — Mon, 18 Nov 2024 18:00:00 +0000

In may 2023, Docker announced a new command line docker init, that could simplify your life when creating new projects.

It is aimed to quickly generate Dockerfile and Docker-compose file for your projects, following the best practices pushed by the editor himself.

Let's see how it works, with a simple example, running on Flask.

Create the app

Create a my-api.py file :

#!/usr/bin/python3
from flask import Flask, jsonify

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "API is running..."

@app.route("/users", methods=["GET"])
def caracters():
    return jsonify({
        "users": [
            "Luke",
            "Leia",
            "Han",
            "Chewi",
            "Darth",
            "Obi",
        ]
    })

if __name__ == '__main__':
    app.run(port=3000, debug=True)

And a requirements.txt file :

flask>=2.2.2
gunicorn

Generate the files

Execute docker init and answer the questions.

It will generate you the following files :

Dockerfile - The instructions to build your image

# syntax=docker/dockerfile:1

# Comments are provided throughout this file to help you get started.
# If you need more help, visit the Dockerfile reference guide at
# https://docs.docker.com/go/dockerfile-reference/

# Want to help us make this template better? Share your feedback here: https://forms.gle/ybq9Krt8jtBL3iCk7

ARG PYTHON_VERSION=3.12.4
FROM python:${PYTHON_VERSION}-slim as base

# Prevents Python from writing pyc files.
ENV PYTHONDONTWRITEBYTECODE=1

# Keeps Python from buffering stdout and stderr to avoid situations where
# the application crashes without emitting any logs due to buffering.
ENV PYTHONUNBUFFERED=1

WORKDIR /app

# Create a non-privileged user that the app will run under.
# See https://docs.docker.com/go/dockerfile-user-best-practices/
ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    appuser

# Download dependencies as a separate step to take advantage of Docker's caching.
# Leverage a cache mount to /root/.cache/pip to speed up subsequent builds.
# Leverage a bind mount to requirements.txt to avoid having to copy them into
# into this layer.
RUN --mount=type=cache,target=/root/.cache/pip \
    --mount=type=bind,source=requirements.txt,target=requirements.txt \
    python -m pip install -r requirements.txt

# Switch to the non-privileged user to run the application.
USER appuser

# Copy the source code into the container.
COPY . .

# Expose the port that the application listens on.
EXPOSE 3000

# Run the application.
CMD gunicorn 'my-api:app' --bind=0.0.0.0:3000

.dockerignore - The files you don’t want to copy to your container

# Include any files or directories that you don't want to be copied to your
# container here (e.g., local build artifacts, temporary files, etc.).
#
# For more help, visit the .dockerignore file reference guide at
# https://docs.docker.com/go/build-context-dockerignore/

**/.DS_Store
**/__pycache__
**/.venv
**/.classpath
**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm
**/bin
**/charts
**/docker-compose*
**/compose.y*ml
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
LICENSE
README.md

README.Docker.md - The Readme associated to Docker elements

### Building and running your application

When you're ready, start your application by running:
`docker compose up --build`.

Your application will be available at http://localhost:3000.

### Deploying your application to the cloud

First, build your image, e.g.: `docker build -t myapp .`.
If your cloud uses a different CPU architecture than your development
machine (e.g., you are on a Mac M1 and your cloud provider is amd64),
you'll want to build the image for that platform, e.g.:
`docker build --platform=linux/amd64 -t myapp .`.

Then, push it to your registry, e.g. `docker push myregistry.com/myapp`.

Consult Docker's [getting started](https://docs.docker.com/go/get-started-sharing/)
docs for more detail on building and pushing.

### References
* [Docker's Python guide](https://docs.docker.com/language/python/)

compose.yaml - The file to execute Docker-compose

# Comments are provided throughout this file to help you get started.
# If you need more help, visit the Docker Compose reference guide at
# https://docs.docker.com/go/compose-spec-reference/

# Here the instructions define your application as a service called "server".
# This service is built from the Dockerfile in the current directory.
# You can add other services your application may depend on here, such as a
# database or a cache. For examples, see the Awesome Compose repository:
# https://github.com/docker/awesome-compose
services:
  server:
    build:
      context: .
    ports:
      - 3000:3000

# The commented out section below is an example of how to define a PostgreSQL
# database that your application can use. `depends_on` tells Docker Compose to
# start the database before your application. The `db-data` volume persists the
# database data between container restarts. The `db-password` secret is used
# to set the database password. You must create `db/password.txt` and add
# a password of your choosing to it before running `docker compose up`.
#     depends_on:
#       db:
#         condition: service_healthy
#   db:
#     image: postgres
#     restart: always
#     user: postgres
#     secrets:
#       - db-password
#     volumes:
#       - db-data:/var/lib/postgresql/data
#     environment:
#       - POSTGRES_DB=example
#       - POSTGRES_PASSWORD_FILE=/run/secrets/db-password
#     expose:
#       - 5432
#     healthcheck:
#       test: [ "CMD", "pg_isready" ]
#       interval: 10s
#       timeout: 5s
#       retries: 5
# volumes:
#   db-data:
# secrets:
#   db-password:
#     file: db/password.txt

Now you can launch your app

Directly on your machine :

pip install flask --user
flask --app my-api --debug run -p 3000

With Docker :

docker build -t my-api .
docker run -d -p 3000:3000 --name my-api my-api

With Docker-compose :

docker init
docker compose up --build

What about Podman ?

Nowadays, we can’t talk about Docker without mentioning it’s alternative Podman.

Both provide very similar commands overall, but the Init command is an exception.

Podman-init has a different purpose : it creates a new container from an image and prepares it to run as a Podman container.

Conclusion

Docker-init is a handy little utility integrated into the command-line.

Consider using it when you need to create new projects (or update old ones), it can save you a few precious minutes, while helping you to be compliant by following best practices.

Create your K3S lab on Google Cloud

Jérôme Dx — Mon, 21 Oct 2024 15:00:00 +0000

If you study Kubernetes, it may interest you to have a little laboratory to experiment the concepts you learned.

You can use a local solution, like Kind, fast and easy to handle, but it may be interesting too to make it work on a Cloud provider, closer to production context.

Our choices

Google Cloud

Google Cloud is the only one among the major providers to offer a free tier for life (unless the terms and conditions change, which remains to be seen, but for now we can enjoy it).

Among the services eligible for the Free Tier, here are those which will interest us :

Compute Engine : 1 non-preemptible e2-micro VM instance in South Carolina (us-east1), the region closest to France
Cloud Storage : 5 GB-months of regional storage (us-east1) per month, which is more than enough storage capacity for a homelab
Artifact Registry : 0.5 GB storage per month
Secret Manager : 6 active secret versions per month
Cloud Logging : Free monthly logging allotment
Cloud Monitoring : Free monthly metrics allotment
Cloud Shell : Free access to Cloud Shell, including 5 GB of persistent disk storage

There are 3 regions eligible for this free tier but living in France, I personally chose the closest geographically : us-east1.

In any case, the prices are interesting on this provider, we will see that we can also use spot instances to reduce costs.

This lab will also be an opportunity to improve your skills on Google Cloud.

K3S

K3S is a Kubernetes distribution made by Rancher, made to be as lightweight as possible while being compatible with Kubernetes production standards.

This is the perfect tool for our lab, because we want to have the lightest instances possible to keep costs down, while having the ability to test production concepts.

Even GKE (Google Kubernetes Engine) will cost you at least around $70 per month.

With K3S you can simply setup an e2-micro instance, with the free-tier (but it will be very limited).

If you want more capacity, you can create an e2-small instance, which will cost you less than $10 per month (with spot instance).

K3S will still be interesting for all the usecases where you want to reduce the size of the instances.

OpenTofu

If you haven't been following the news, Hashicorp has changed the license of Terraform and has had some issues with the community.

OpenTofu is a fork of Terraform, aimed to remain open-source and the changelog is driven by the community.

Currently, at version 1.8, OpenTofu is slightly the same as Terraform, just change the binary, you can install it with Tenv (differences will appear as the respective roadmaps will evolve).

This lab is therefore an opportunity to familiarize yourself with the use of this tool.

Setting up the lab

Now we will see how to implement this solution, the usecase is to deploy the resources on Google Cloud, create the K3S cluster on the VM, then create an API accessible from outside.

Create the Google Cloud project

First, if you haven't already, you need to create a Google Cloud project.

At the creation of the project, you can activate a 90-day $300 Free Trial offer, on a large panel of services, with some limitations though.

After this period ends, you will still be eligible for the free tier we already mentioned.

Deploy the infrastructure

Here are the resources you will create with OpenTofu :

// Compute
// ----------------------------------

// The instance for K3S
resource "google_compute_instance" "k3s" {
  name         = "k3s-vm-1"
  machine_type = "e2-small" # This instance will have 2 Gb of RAM
  zone         = var.zone

  tags = ["web"]

  // Set the boot disk and the image (10 Gb)
  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-12"
      size  = 10
    }
  }

    // Configuration to be a Spot Instance, to reduce costs
  scheduling {
    preemptible                 = false
    automatic_restart           = true
    provisioning_model          = "SPOT"
    instance_termination_action = "STOP"
  }

  // attach a disk for K3S
  attached_disk {
    source      = google_compute_disk.k3s_disk.id
    device_name = "k3s-disk"
  }

  network_interface {
    network = "default"

    access_config {
      // Ephemeral public IP
    }
  }

  labels = {
    env       = var.env
    region    = var.region
    app       = var.app_name
    sensitive = "false"
  }

  metadata_startup_script   = file("scripts/k3s-vm-startup.sh")
  allow_stopping_for_update = true
}

// Firewall
// ----------------------------------
resource "google_compute_firewall" "allow_http" {
  name    = "allow-http"
  network = "default"

  allow {
    protocol = "tcp"
    ports    = [
      "80", "443", // http/https
      "30080"      // ports opened to access the API via NodePort
    ]
  }

  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["web"]
}

// Storage
// ----------------------------------

// The disk attached to the instance (15 Gb)
resource "google_compute_disk" "k3s_disk" {
  name = "k3s-disk"
  size = 15
  type = "pd-standard"
  zone = var.zone
}

// The bucket where you can store other data
resource "google_storage_bucket" "k3s-storage" {
  name     = var.bucket_name
  location = var.region

  labels = {
    env       = var.env
    region    = var.region
    app       = var.app_name
    sensitive = "false"
  }
}

// Registry
// ----------------------------------

// The Artifact Registry repository for our app
resource "google_artifact_registry_repository" "app-repo" {
  location      = "us-east1"
  repository_id = "app-repo"
  description   = "App Docker repository"
  format        = "DOCKER"

  docker_config {
    immutable_tags = true
  }
}

// Env vars
// ----------------------------------

variable "env" {
  type        = string
  default     = "dev"
  description = "Environment"
}

variable "region" {
  type        = string
  default     = "us-east1"
  description = "GCP Region"
}

variable "zone" {
  type        = string
  default     = "us-east1-a"
  description = "GCP Zone"
}

variable "app_name" {
  type        = string
  default     = "<name-of-your-cluster>"
  description = "Application name"
}

variable "project_name" {
  type        = string
  default     = "<name-of-your-gcp-project>"
  description = "GCP Project name"
}

variable "bucket_name" {
  type        = string
  default     = "<name-of-your-bucket>"
  description = "Bucket name"
}

// Provider
// ----------------------------------

// Connect to the GCP project
provider "google" {
  credentials = file("<my-gcp-creds>.json")
  project     = var.project_name
  region      = var.region
  zone        = var.zone
}

terraform {
  # Use a shared bucket (wich allows collaborative work)
  backend "gcs" {
    bucket      = "<my-bucket-for-states>"
    prefix      = "k3s-infra"
  }

  // Set versions
  required_version = ">=1.8.0"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">=4.0.0"
    }
  }
}

The startup script (in “scripts/k3s-vm-startup.sh”), which will install K3S automatically :

#!/bin/bash

# Format the disk if not already formatted
if ! lsblk | grep -q "/mnt/disks/k3s"; then
    mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/disk/by-id/google-k3s-disk
    mkdir -p /mnt/disks/k3s
    mount -o discard,defaults /dev/disk/by-id/google-k3s-disk /mnt/disks/k3s
    chmod a+w /mnt/disks/k3s
fi

# ensure only run once
if [[ -f /etc/startup_was_launched ]]; then exit 0; fi
touch /etc/startup_was_launched

# apt install
apt update
apt install -y ncdu htop

# helm install
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
/bin/bash get_helm.sh

# bashrc config
rc=/root/.bashrc
echo "alias l='ls -lah'" >> $rc
echo "alias ll='ls -lh'" >> $rc
echo "alias k=kubectl" >> $rc
echo "export dry='--dry-run=client'" >> $rc
echo "export o='-oyaml'" >> $rc

# Install k3s and configure it to use the persistent disk for data storage
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--data-dir /mnt/disks/k3s" sh -

Create a ServiceAccount to connect to GCP with OpenTofu, restricted to this services :

(For least privileges compliance, you must restrict the elements with IAM conditions)

Artifact Registry Administrator
Artifact Registry Repository Administrator
Cloud Functions Admin
Compute Admin
Compute Instance Admin
Compute Network Admin
Compute Security Admin
Secret Manager Admin
Service Account User
Storage Admin

Then, the commands to create the infrastructure :

# OpenTofu setup
tofu init -get=true -upgrade
tofu workspace new dev
tofu workspace select dev

# Plan (to preview what will be changed)
tofu plan

# Apply (to create the infrastructure described in the IaC code)
tofu apply

Build the app

Create a Dockerfile for your app, we will use the HttpBin API which allows to test all the request we can make to a Rest API :

FROM python:3.12-slim

# Install dependencies
RUN pip install --no-cache-dir gunicorn httpbin

# Expose the application port
EXPOSE 80

# Launch the application
CMD ["gunicorn", "-b", "0.0.0.0:80", "httpbin:app"]

Make the build and push the image to Artifact Registry

# Build
docker build -t httpbin .

# Push to the registry
gcloud auth configure-docker us-east1-docker.pkg.dev
docker tag httpbin us-east1-docker.pkg.dev/<my-project>/app-repo/httpbin:v1
docker push us-east1-docker.pkg.dev/<my-project>/app-repo/httpbin:v1

Set the secret to connect to the registry

Create a Service Account, with the right “Artifact Registry Reader”.

It will allow you to pull the image from Kubernetes.

First, connect to your instance :

gcloud compute ssh --zone "us-east1-a" "k3s-vm-1" --project "<my-project>”

Then, store the credentials on Kubernetes like this :

echo -n "<json_value>" > registry_key.json

k create secret docker-registry artifact-read \
    --docker-server=us-east1-docker.pkg.dev \
    --docker-username=_json_key \
    --docker-password="$(cat registry_key.json)" \
    --docker-email=valid-email@example.com

Deploy the app on Kubernetes

Here is the code to deploy your app :

# The deployment that will pull the image on Artifact Registry
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      imagePullSecrets:
      - name: artifact-read
      containers:
      - name: httpbin
        image: us-east1-docker.pkg.dev/<my-project>/app-repo/httpbin:v1
        ports:
        - containerPort: 80
---
# The Service, configured as a NodePort to allow external access on some ports
apiVersion: v1
kind: Service
metadata:
  name: httpbin-service
spec:
  type: NodePort
  ports:
    - port: 80
      nodePort: 30080
  selector:
    app: httpbin

Then deploy it :

# deploy the app
k apply -f httpbin.yaml

# Ensure the app is running
k get po -w

Access the API

Once it’s running, you can access the API inside the VM :

curl -I http://localhost:30080/get
# HTTP/1.1 200 OK

Or outside, from your local machine :

curl -I http://<my-ephemeral-ip>:30080/get
# HTTP/1.1 200 OK

Conclusion

We’ve just seen how to deploy quickly and with minimal costs a modern lightweight Kubernetes on your Google Cloud project.

There will be other workshops that may be interesting to improve this lab, like :

Add a Traefik reverse proxy to access to several apps
Add Cloudflare to protect exposed urls
Set a multi-masters, multi-workers pattern
Set a Backup/Restore strategy
Setup Cilium
Add monitoring and alerting tools

And many other things.

Looking forward to exploring these different topics with you next time.

Managing Cloud Costs and Security with Tailwarden

Jérôme Dx — Mon, 09 Sep 2024 15:00:00 +0000

⚠️ Warning

This article is kept as an archive;
unfortunately, the development of this service has been discontinued.

When working on the cloud, the question of costs is essential.

Tailwarden is a tool designed to help you have a global supervision of your costs and also supports you on the security part.

It supports the 3 main cloud providers (AWS, Google Cloud and Azure), it will be all the more effective in a multi-account and hybrid strategy.

I will give you a demonstration of the different functionalities that the tool will offer you.

Connect your cloud accounts

At the beginning, you have access to a "Getting started".

You will be able to connect your different accounts, in my case an AWS account and a GCP account.

If you want detailled informations, you have to enable the cost insight feature :

The main dashboard

You will have direct access to the main dashboard, which gives you an overall view of your costs per month.

You will be able to classify them by cloud provider, regions and resource types.

The assets inventory

The inventory is the heart of Tailwarden, that’s where you will find all the resources handled by the software. You can consult, classify them, know the cost of each one and export them for your needs.

Create your first custom view

From the inventory, you can apply filters, to obtain Custom views, which will allow you to organize the vision of your infrastructure, according to the criteria that interest you.

In the following example, I simply display the list of S3 buckets :

Tagging

Identify untagged resources

The Tags audit page will allow you to explore the tags of your resources.

In order to find your untagged resources, you can create a Custom view by selecting the "Empty tags" filter :

Set a tagging strategy

It is important to establish a tagging strategy, you can follow the Tagging Best Practices shared by Tailwarden.

We can discern 4 main categories:

Technical tags
Automation tags
Business tags
Security tags

Consider that your fleet will potentially be hybrid one day, it is better to put provider-agnostic tags.

For the demonstration, we will choose a very basic tagging strategy, with this tags :

Env (Dev/Prod/Shared)
Region
Sensitive (true/false)
Project (null by default)

We can set a policy to show which resources are in compliance or not :

Apply and monitor your tagging strategy

You can modify directly the provider’s tags, or define virtual tags inside Tailwarden.

Virtual tags allow you to quickly edit your resources in groups and to comply.

You will then have the possibility to either synchronize the provider tags, or to apply a different strategy within the provider.

Compliance is monitored and you have the possibility to put alerting, in case there are drifts.

Enable Cost insights

You must setup CUR (Cost and Usage Report), to activate Cost Insights :

This will allow for more cost detail, more accurately for your resources.

Create your first Cost report

In the Cost Report section, you will have a global view on your costs, for each provider :

Set your first Budget alert

From a view in the inventory, you have the possibity to setup notifications by mail, on different triggers :

For example, you can set an email notification when your overall costs exceed an amount you decide, withe the Budget alert:

View Compliance insights

In the Risk Assessment section, you can consult different compliance frameworks to apply on your resources :

AWS CIS (Center for Internet Security) v1.4.0 : The security framework from AWS
GPDR (General Data Protection Regulation) : The european privacy data protection
PCI DSS (Payment Card Industry Data Security Standard) : A framework you must follow if you host credit card informations

Custom dashboards

In the Reports sections, you can set specific dashboards for your needs, you can totally customize your dashboards and widgets, but you can also select pre-defined templates.

For instance, there is an Invoice Breakdown dashboard, to see the costs specific to each resource :

Pricing

Tailwarden is a managed service aimed at businesses, it offers a starter package at $500/month, with additional costs if there are more than 3 cloud accounts.

It can also offer more support for larger companies.

Another option is a similar product offered by Tailwarden, which is open source. It will offer a few different options and the hosting part will be up to you. This is Komiser, whose code is available on Github, and which provides a Docker image.

What’s next

To manage your costs and monitor the security of your infrastructure, Tailwarden is a good option that will give you a global vision very easily.

It is a young product but also very active and promising, with a public roadmap available, which promises new features in particular on the Anomaly detection and Custom dashboard parts.

A state of AI in 2024

Jérôme Dx — Mon, 15 Jul 2024 14:00:00 +0000

If I tell you that about 2 years ago, GenAI arrived with ChatGPT and that now we almost only hear about Artificial Intelligence, in every way, I probably don't tell you much.

I would therefore like, in this article, to bring clarity to the subject, by summarizing the implications of AI today, what are the current issues that we are facing, but also, the concrete and useful applications from which we can benefit.

An abstract definition

AI (Articial Intelligence) is a field of computer science that focuses on creating systems capable of performing tasks that typically require human intelligence. These tasks include learning from experience, recognizing patterns, solving problems, understanding natural language and making decisions.

Pay attention to one thing, it is important to realize that this is a very generic expression and that it is nothing more or less about computer, algorithmic techniques which are actually used, far from replacing the functioning of a human brain.

In other words, what we call “AI” depends entirely on our perception of it, and should rather be called “algorithmic techniques”. So much so that Luc Julia, who created Siri, came to say that “There is no such thing as AI”.

A technical definition

Systems commonly associated to AI are designed to learn from data, adapt to new inputs, and improve over time. This key components and subfields include :

Machine Learning (ML): A subset of AI that involves training algorithms to recognize patterns and make decisions based on data. ML models are built using statistical techniques to enable systems to improve their performance on a given task with more data over time.
Deep Learning (DL): A specialized branch of ML that employs neural networks with many layers (hence "deep") to model complex patterns in large datasets. DL is particularly effective in tasks such as image and speech recognition.
Large Language Models (LLMs): These are deep learning models that are trained on vast amounts of text data to understand and generate human language. Examples include GPT (Generative Pre-trained Transformer) and BERT (Bidirectional Encoder Representations from Transformers).
Generative AI (GenAI): A type of AI focused on creating new content, such as text, images, or music, that is indistinguishable from human-generated content. This includes techniques like Generative Adversarial Networks (GANs) and autoregressive models like GPT.
Retrieval-Augmented Generation (RAG): A hybrid approach combining retrieval-based and generative techniques. RAG systems first retrieve relevant documents or information from a large corpus and then generate responses based on the retrieved content. This approach enhances the accuracy and relevance of generated outputs.

In essence, AI leverages these advanced techniques to build systems that can perform sophisticated tasks, ranging from data analysis and prediction to natural language understanding and content generation.

Two things should be noted: on the one hand, these are only statistical and analysis tools, which go through large sets of data, very far from the functioning of a human brain. The other thing is that, for the end user, what is important is still the perception, to qualify as AI, so these techniques will not necessarily be used in the product that is sold as such.

Just another buzzword ?

It has become the latest hype in IT, GenAI has propelled AI onto the podium, as have blockchain, VR headsets, or even 3D TVs, which, as we know, left the front of the stage as they came.

It is also good to remember that AI is a subject as old as computing itself, periods of craze followed by disinterest, called "AI winters", have been regularly observed, the first dating from 1966.

The hype is such that in recent times, the term "AI" has been attached to most computer products, whether it has any real meaning or not, sometimes in a truly ridiculous or inappropriate way. Each application wants its "AI feature".

The same goes for computer conferences that were twisted into being on the topic of AI, even though the real topics were completely different.

One truth remains, this enthusiasm unlocks research funding, which allows us to believe that we will be able to see real progress emerge in the future.

A threat ?

I don't know if you've noticed, but our thoughts often move much faster than our ability to act.

We are indeed witnessing progress currently, as well as an amplifying marketing and journalistic frenzy, of which we must not be fooled.

I'm not going to dwell too much on the subject, but we can discern worrying scenarios, which are the loss of jobs and loss of control.

Loss of jobs

A widely talked about study from the IMF indicates that 40% of jobs would be affected by AI. These predictions should be taken with a grain of salt for several reasons.

On the one hand, certain jobs were assigned impressively quickly, such as graphic designers or developers. These professions generally know how to adapt to new technological developments and take advantage of them to gain productivity, we cannot therefore talk about job losses. Besides, in the next section, I will talk about the tools that I use daily as a developer.

On the other hand, it must be taken into account that the development of new working methods, as all industrial revolutions have demonstrated, have not led all of humanity to idleness, but have contributed to the creation of new jobs, whether qualified or not. Even if unemployment issues are very real, they are also nuanced and contextual.

We must not hide our faces either, we will observe changes due to paradigm shifts induced by AI, such as recently the elimination of jobs in Duolingo's translator teams, the deterioration of the DeviantArt website, AI Crawlers, or many other examples.

What is certain is that jobs as we know them will evolve and the skills needed will be different, which concerns current and especially future generations.

Loss of control

AI may take control of humans, betray them or even destroy them, that's what Skynet, HAL 9000, I, Robot, or globally Hollywood and Pop Culture told us. We can also call it the "Hollywood AI".

These are only extremely unlikely scenarios. For this, the machine would have to truly emancipate itself from human command, have sufficient power of action, have a motive for such actions and above all work completely differently than the tools we develop today.

For the moment, all that's happening are clickbait publications, such as, for example, that Google AIs are plotting among themselves in an unknown language, or that a tool is working to destroy humanity.

It is important to be alerted to the threats that these new tools may have (we will talk about them again, particularly on privacy and ecology), but we must also put things into perspective and carefully check the sources of the informations we consume, to demystify things and understand them.

Individual usage

The opening of access to the general public to ChatGPT at the beginning of 2023 was successful and its popularization was impressive.

The general public very quickly understood the services that it would provide them:

Help with content generation (homework, articles, creations of all kinds)
Search assistance, which replaces the Google engine
Help with problem solving

This use has become a new common model, for example, for this search, it is simpler to ask ChatGPT to give me the solution, rather than searching from page to page to find my answer (and that's very complete) :

Google, which has long been at the forefront in the field of AI, suddenly found itself shaken up by the acquisition of ChatGPT by Microsoft. Google therefore finds itself in the situation of being confronted on the front line by these GenAI tools which aim to replace the classic Google search page, and it is not in a strong position.

Now, Google offers Gemini (its ChatGPT competitor) on its home page and Bing offers Copilot (which uses the ChatGPT engine):

Many tools on GenAI have emerged recently and are referenced on the futuretools site.

The next developments revolve around improving the engine: the completeness of its dataset, the speed of execution, as well as the recent update of the engine training and its internet access.

The other development is to be multimodal: this affects images, music and even video.

Sora, OpenAI's tool for generating video, unveiled a fairly impressive video in Japan, even if some imperfections were detected.

These tools are very practical and impressive, but require human validation (or even editing) and require new skills to generate the most effective prompt possible.

Like any tool, it can also be misused, which can lead to the deterioration of some content available online. It is up to us to also be vigilant about the quality of the content we consume.

Industry usage

For developers, Microsoft has been offering Github Copilot for several years now, which allows assistance with writing code (whether in autocomplete or via a prompt).

GenAI is also very useful for debugging phases, because it gives food for thought, hence the reaction of Stack Overflow), which also offered an AI service.

For Cloud architecture usage, today the challenge is no longer to build your own Machine Learning or LLM models, given that they are available off the shelf by Cloud providers.

Each provider therefore offers a range of services to support the use of these technologies.

For the moment Google and Microsoft are in the lead, AWS also offers a service called Bedrock that can use different Foundation Models, including Mistral AI.

AWS also offers Amazon Q, a tool more focused on the AIOps approach, which aims to integrate AI into the operational part.

It’s interesting to see that AWS, which has always been at the forefront of the market, seems to be behind its competitors on GenAI. We don't know how it will evolve, but they could surprise us, they are investing actively in this domain.

At a time when we are talking about sovereign cloud and GDPR in Europe, GenAI represents an additional constraint in terms of data protection and will probably not be validated in all contexts (because usage means giving your data to the cloud provider that provides the service).

Despite everything, GAFAM are at the forefront but today a lot of brands are trying to associate their image as being at the cutting edge of AI, this extends to automobile or even cycling industry.

Limitations

The technology has developed well, but creates new problems inherent in the way it works and requires increased supervision:

Hallucinations: To answer correctly, the model can very well invent informations from scratch with breathtaking confidence. This is why it is essential to scrupulously recheck everything before validating and ensuring that you have sufficient expertise for the area being addressed.
Bias: For example in image generation, you type: “Show me a CEO”: the generator will only output images of tall brown men with white skin. These are clichés that can become even more ingrained in our way of thinking, instead of encouraging diversity.
Reasoning problems: The model does not have our way of thinking and is capable of making grotesque errors, certain examples demonstrate this: “Making a sentence without the letter e", “Give a positive prime number after 2”, etc. The engine returns logical errors with the greatest confidence.

These problems are however corrected with each version, but will never be corrected completely.

We must also not forget the problems already inherent in computing, which are greatly amplified by AI models and constitute, more than anything else, the real problems caused by AI:

Privacy: Data models recover a large amount of information, of which we do not necessarily have the capacity to verify whether they are respectful of privacy or not, we have to trust the software provider. There is a particular topic with Meta about the use of personal information to train its models.
Ecology: Climate issues are more than ever at the forefront, at a time when we could instead advocate a form of energy saving, AI models consume a lot of resources: even though this information is kept private, we can clearly imagine (confirmed among others by a Sam Altman declaration) that this represents astronomical quantities of GPUs running constantly (which makes Nvidia happy) and that it’s only growing, we can unfortunately no longer count on Moore's law to compensate.

Regulation

Failing to be at the forefront in the field of creation, Europe has taken the step of being at the forefront in the field of regulation with the AI Act. This is nevertheless a major advantage, because the problems posed are very real and complex in the regulation of such tools.

To conclude

The IT market has found itself extremely shaken up recently: all software wants to release its AI feature, IT specialists suddenly have to be specialists in AI and the big players are investing colossal sums to corner the market.

In this context, it is interesting to adapt to take advantage of these technologies, to always marvel at the progress and the help it brings us, but also to be aware of the issues (particularly on the privacy and ecology side), hence the importance of regulation.

Keep in mind that the use of these technologies requires reinforced supervision in order to always produce quality work.

The big development that makes many people dream, but which is probably not for tomorrow, contrary to what Elon Musk said, is the AGI (Artificial General Intelligence), maybe that will be the next breakthrough, or maybe we will just get tired of AI, saturated from having talked about it too much, until the next real update.

In any case, it is important to follow the news with full knowledge of the facts, by selecting your sources carefully and by following people who really practice AI and can talk about it without mystification, as do very well Yann Le Cun, Aurélie Jean and Luc Julia.

More progress is still to come and it is very likely that I will write other articles on these algorithms which work on data sets, which some have decided to call “AI”.

Tech radar: Keep an eye on the technology landscape

Jérôme Dx — Mon, 17 Jun 2024 14:00:00 +0000

If you've been working in technology for a while, you've probably noticed that there are so many technologies out there and things are constantly evolving.

My goal here is to give you some keys to not miss essential information and find your way in the technological jungle, while having the ability to completely disconnect once your working time is over.

Shared tech radars

Some companies share the tech radar they created, you can consult them whenever you want :

ThoughtWorks - A global technology company
Zalando Engineering - Company specializing in the sale of shoes and clothing
Theodo Cloud - DevSecOps Experts
Devoteam - Technology consulting Agency
Ippon - Another Technology consulting Agency

The main categories for a Tech radar are :

Adopt : Approaches in which we have great confidence
Trial : Approaches we've seen work successfully
Assess : Approaches that are promising and have clear potential added value
Hold : Approaches that are not recommended for use on new projects

We can also mention Digital.ai which maintains a Periodic table of DevSecOps tools allowing you to explore technologies around devops in an original and practical way.

Build your own tech radar

For targeted and punctual needs for you or your company, you can create an Excel sheet to help you make choices, or adopt the Tech Radar approach. Some companies share their method and tools for building your own Tech Radar, like ThoughtWorks and Zalando.

Be careful though, if you build your own knowledge base, it will be a real effort to update it, given the speed at which technologies evolve today.

Software dictionaries and comparators

The common use case is that you hear about a new technology, and very quickly need to understand what it does, what are the alternatives and where it fits into the big picture.

Luckily, there are community platforms which reference software and allow you to find your way around, here is a selection :

Alternativeto - References all types of software on all OS : Windows, Mac, Linux, Android, etc.
Stackshare - Aimed for companies building their technical stack
Trust Radius - Software database with comparisons and reviews
Product Hunt - To discover and share new tech products
Cloud Native Landscape - Tools referenced by the CNCF, concerning Kubernetes and containers
Future Tools - A curated list of new AI tools

Obviously your favorite search engine or AI assistant are also tools of choice for finding information on technologies you want to discover. One tip to find alternatives for a product is to type on Google : <product> vs and the autocomplete feature will help you diligently.

Trends

To help you to make decisions, Google Trends shows you tendencies about the products, compared to others. It is very useful to have it in your toolbox.

We can also cite Gartner which provides analyzes and predictions, some of which are available for free.

Obsolescence management

Once you have adopted the tools in your stack, you must monitor the life of the projects in order to always keep them up to date, with regularly scheduled updates.

The End-of-life website is made for this :

Learning roadmaps

if you want to master all the technologies that make up your profession, there is the roadmap.sh site, which gives you learning paths for particular areas (Devops, Kubernetes, Python, etc.)

Tech blogs & feeds aggregation

Some people enjoy sharing their knowledge on technical areas, blogs remain accessible and effective media.
Feedly or Inoreader will allow you to follow the RSS feeds of these websites and consult them at your leisure, so you can follow the specific areas that interest you.

Also think about the Dev.to platform which allows developers to easily share on their topics.

Other preferred media at the moment are Podcasts and YouTube, that have a good base of quality content, and have the advantage of being able to be listened to in transport or during a running session.

Newsletters

Newsletters can remain a good option for keeping up to date without searching for information yourself.

The TLDR Newsletter is an interesting option if you don't have a lot of time. It summarizes new developments in general, or in particular areas: Devops, AI, etc.

This allows you to optimize your time and get the essentials without getting too lost.

Latest trends and interactions

If you have a little more time to spare, you can connect on social networks.

X (ex-Twitter) is the long-standing platform for getting and sharing early access to information, with a large user base and a uniquely fun feel. It's also a great way to stay up-to-date with tech events and conferences.

You can favor other alternatives like Mastodon, or Bluesky, which has recently made a breakthrough. Although the user base is still smaller than that of X, these are functional and promising microblogging platforms.

LinkedIn is also used to share tech news these days.

Reddit is a platform with a strong user base, which provides exclusive news and technical knowledge.

To share about specific technologies, Slack or Discord channels are quite common and effective ways to interact with the community.

To conclude

We've seen tools and methodologies to keep tabs on what's happening in tech news and always have an overview of the tech landscape so you know how to navigate it.

It's now up to you to take advantage of these tools in order to stay up to date with current events and different tools while maintaining a good life balance on a personal level.

Enjoy your technological exploration, which is an exciting journey and an endless source of inspiration.

Github Actions to deploy your Terraform code

Jérôme Dx — Tue, 21 May 2024 09:41:28 +0000

If you have chosen to host some of your codebases on Github, you should know that there is a built-in tool allowing you to implement CI/CD workflows, called Github Actions.

My goal in this article is to show you that with just a few clicks and writing yaml, you can properly deploy your Terraform code on your favorite cloud provider (in this example we will use Google Cloud).

Why CI/CD

In short, the main advantages that I see of going through a CI/CD pattern (Continuous Integration / Continuous Deployment), rather than deploying the code directly from your computer, are :

Maintenance : Your deployment code will be reproducible and versioned
Traceability : You will have a history of different deployments and past errors
Teamwork : Deployment will be accessible to all those included in the project, without complicated configuration

For a more exhaustive explanation, I refer you to the reference which is the 12factor app.

Why Github Actions

Github Actions has many competitors in its category that allow you to run all kinds of code running on containers, such as Gitlab, Jenkins, CircleCI, etc.

There are also more specialized tools in deploying IaC code, such as: HCP Terraform, Atlantis, Terramate, etc.

My choice fell on Github Actions, aiming to have the lightest footprint possible.

Since my code is already on the Github platform, this gives me the ability to execute code, without having to connect to an additional tool. The possibilities are extensive, I can run all kinds of containers, whether for infrastructure or not.

It is also a tool with a strong community, which can be frequently encountered on consulting missions.

Pricing

Considering the billing documention, for personal use, you can use it free of charge within certain limitations per month for private repositories (2000 minutes of build and 500 Mo of package storage, by month). You also have the possibility to connect your own agents in addition.

Good news is that Github encourages open-source by offering unlimited use for publicly exposed repositories.

If you want to extend the possibilities of your personal account, you can take Github Pro, for $4/month.

If the company you work for has subscribed to Github, you probably benefit from a more substantial offer with additional features (GitHub Team or GitHub Enterprise).

Let’s do the demo

In our example, I only selected the basic features available in the free offer, which can therefore be used in all contexts.

Here is the Terraform code you can use to create a Cloud function on Google Cloud :

# main.tf
resource "**google_storage_bucket_object**" "object" {
  name   = "functions/test1.zip"
  bucket = var.bucket_name
  source = "./functions/test1.zip" # Local path to the zipped function source code
}

resource "google_cloudfunctions2_function" "function" {
  name        = "mycorp-test1"
  location    = var.region
  description = "Test1 function"

  build_config {
    runtime     = "python312"
    entry_point = "main"
    source {
      storage_source {
        bucket = var.bucket_name
        object = google_storage_bucket_object.object.name
      }
    }
  }

  service_config {
    max_instance_count = 1
    available_memory   = "256M"
    timeout_seconds    = 60
  }
}

# output.tf
output "function_uri" {
  value = google_cloudfunctions2_function.function.service_config[0].uri
}

# provider.tf
provider "google" {
  credentials = file("credentials.json")

  project = var.project_name
  region  = var.region
  zone    = var.zone
}

terraform {
  backend "gcs" {
    bucket      = "infra-mycorp-terraform-states"
    prefix      = "gha-demo"
    credentials = "credentials.json"
  }
  required_version = ">=1.7.0"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">=4.0.0"
    }
  }
}

# variables.tf
variable "region" {
  type        = string
  default     = "europe-west1"
  description = "GCP Region"
}

variable "zone" {
  type        = string
  default     = "europe-west1-c"
  description = "GCP Zone"
}

variable "project_name" {
  type        = string
  default     = "infra-mycorp"
  description = "GCP Project Name"
}

variable "bucket_name" {
  type        = string
  default     = "mycorp-packages"
  description = "Bucket name"
}

You may also generate a json key for your Service Account.

Here is the Python code you can use to create an example package for your Cloud Function, to store on a Cloud Storage bucket :

import functions_framework

@functions_framework.http
def main(request):
    """HTTP Cloud Function.
    Args:
        request (flask.Request): The request object.
        <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
    """
    request_json = request.get_json(silent=True)
    request_args = request.args

    if request_json and 'name' in request_json:
        name = request_json['name']
    elif request_args and 'name' in request_args:
        name = request_args['name']
    else:
        name = 'World'
    return 'Hello {}!'.format(name)

And here the Github Actions workflows to execute your Terraform code :

# .github/workflows/1_plan.yml
name: 1. Plan
run-name: Run Workflow ${{ github.run_id }} 🚀
on:
  workflow_dispatch:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo "🚀 Plan launched"
      - name: Install packages
        run: |
          sudo apt-get update
          sudo apt-get install -y build-essential python3
      - name: Install Homebrew
        run: |
          /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
          rc=/tmp/rcfile && touch $rc
          echo 'eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)' >> $rc
      - name: Install Terraform
        run: |
          source /tmp/rcfile
          brew install tfenv
          tfenv install 1.8.3
          tfenv use 1.8.3
          terraform version
      - name: Load credentials
        run: |
          echo -n '${{ secrets.GOOGLE_CREDENTIALS }}' | base64 -d > credentials.json
      - name: Terraform Init
        run: |
          terraform init -get=true -upgrade
          terraform workspace new ${{ vars.WORKSPACE_DEV }} || true
          terraform workspace select ${{ vars.WORKSPACE_DEV }}
      - name: Terraform Format
        run: |
          terraform fmt
      - name: Terraform Plan
        run: |
          terraform plan
      - run: echo "🍏 The job's status is ${{ job.status }}."

# .github/workflows/2_apply.yml
name: 2. Apply
run-name: Run Workflow ${{ github.run_id }} 🚀
on:
  workflow_dispatch:
jobs:
  apply:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo "🚀 Plan launched"
      - name: Install packages
        run: |
          sudo apt-get update
          sudo apt-get install -y build-essential python3
      - name: Install Homebrew
        run: |
          /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
          rc=/tmp/rcfile && touch $rc
          echo 'eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)' >> $rc
      - name: Install Terraform
        run: |
          source /tmp/rcfile
          brew install tfenv
          tfenv install 1.8.3
          tfenv use 1.8.3
          terraform version
      - name: Load credentials
        run: |
          echo -n '${{ secrets.GOOGLE_CREDENTIALS }}' | base64 -d > credentials.json
      - name: Terraform Init
        run: |
          terraform init -get=true -upgrade
          terraform workspace new ${{ vars.WORKSPACE_DEV }} || true
          terraform workspace select ${{ vars.WORKSPACE_DEV }}
      - name: Terraform Apply
        run: |
          terraform apply -auto-approve
      - run: echo "🍏 The job's status is ${{ job.status }}."

# .github/workflows/3_destroy.yml
name: 3. Destroy
run-name: Run Workflow ${{ github.run_id }} 🚀
on:
  workflow_dispatch:
jobs:
  destroy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo "🚀 Plan launched"
      - name: Install packages
        run: |
          sudo apt-get update
          sudo apt-get install -y build-essential python3
      - name: Install Homebrew
        run: |
          /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
          rc=/tmp/rcfile && touch $rc
          echo 'eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)' >> $rc
      - name: Install Terraform
        run: |
          source /tmp/rcfile
          brew install tfenv
          tfenv install 1.8.3
          tfenv use 1.8.3
          terraform version
      - name: Load credentials
        run: |
          echo -n '${{ secrets.GOOGLE_CREDENTIALS }}' | base64 -d > credentials.json
      - name: Terraform Init
        run: |
          terraform init -get=true -upgrade
          terraform workspace new ${{ vars.WORKSPACE_DEV }} || true
          terraform workspace select ${{ vars.WORKSPACE_DEV }}
      - name: Terraform Destroy
        run: |
          terraform apply -destroy -auto-approve
      - run: echo "🍏 The job's status is ${{ job.status }}."

Some explanations on the workflows

Triggers

In the “on:” section, I created a rule to execute the “Plan” workflow, when there is a commit or a pull request on the “main” branch.

I created 3 files (one file for each workflow, inside the “./.github/workflows” folder), with the “workflow_dispatch:” section, we have the possibility to trigger manually an “apply” or a “delete”, once the plan is done (or to relaunch the plan).

Note that I didn’t use the manual approval feature, to set manual validation inside a workflow, because it isn’t available in the free features.

Here is what shows the interface for a basic plan :

Checkout

Ensure you have the step “uses: actions/checkout”, because it is required if you want to have your code inside your container.

Variables & secrets

Go in the Settings of your project, to consult variables and secrets you want to store for your workflow. Then you can call them with the “var.” and “secrets.” prefixes.

There are some restrictions for secrets, once copied, you can’t consult them again from the interface.

Another thing is that you a restricted to a certain number of characters for the value of your secret, a common trick I used is to encode it in base64, and decode it directly in the workflow.

Packaging

I used the “ubuntu-latest” image which is Debian-based (with apt) and commonly available in the Github Actions workers, so it deploys very fast.

I chose to install HomeBrew, because there are a lot of tools I like available on this package manager, including tfenv, which allows me to install any version of Terraform very easily.

Validations and workspaces

The only validation I have installed is the Terraform Format command, which indicates if the code is written according to the recommended formalism. It is of course possible and encouraged to install other validations.

The workspace for Terraform allows you to compartmentalize the code according to different spaces and environments.

To conclude

Here is a very simple way to deploy infrastructure on Google Cloud with a Github free account and some Terraform code.

You can use as you wish and improve according to your preferences and needs.

It is also possible to interact Github Actions with a more specialized tool (like HCP Terraform or Terramate), which could be the subject of a future article.