DEV Community: rndmh3ro

Combining jinja2-cli with jq and environment variables

rndmh3ro — Mon, 30 Jun 2025 18:00:00 +0000

In a current project I’m templating files using Jinja2 with the help of the jinja2-cli. Having used (and taught )Ansible for years I’m quite familar with Jinja2 so this was a natural choice.

The jinja2-cli can receive variables to template by different means. One way is by loading them from a file. This can be json, yaml, toml or even xml.

Suppose I have a json-file called payload.json with the following content:

{
"NAME": "bar",
}

And a jinja2-template called template.j2 like this:

hello {{ NAME }}

Then I can run the following command and it prints the templated contents:

> jinja2 template.j2 payload.json
hello bar

Now for some real world example. I have a slightly more complicated configurtion file I need to template. This configuration file already exists and used the normal jinja variable-syntax with {{ and }}.

I need to load the variables from environment variables.

export NAME=bar
export TENANT=tenant
export BASE_DOMAIN=example.com
export HUB_FQDN=hub.example.com

Since the jinja2-cli can also use environment variables, I could change my template to:

hello {{ env(NAME) }}
tenant is {{ env(TENANT) }}
base domain is {{ env(BASE_DOMAIN) }}
hub fqdn is {{ env(HUB_FQDN) }}

This would load the environment variable NAME into the template.

But I don’t want to change my template, so I need to create a payload.json file with the variables that the jinja2-cli can then load.

How should I create this json-file? I could just write it by hand and use normal bash variable substitution. But that’s not very nice.

The go-to tool for handling json on the command line is jq.

I can use jq to create the json-file like this:

> jq -n \
--arg NAME $NAME \
--arg TENANT "$TENANT" \
--arg SSP_BASE_DOMAIN "$SSP_BASE_DOMAIN" \
--arg HUB_FQDN "$HUB_FQDN" \
'$ARGS.named' > payload.json

The cool part about this is $ARGS.named - this takes all arguments and their vales and prints them. I then write the output into the payload.json file.

Now I can export the variables and run the template:

> jinja2 template.j2 payload.json
hello bar
tenant is tenant
base domain is example.com
hub fqdn is hub.example.com

And I’m done!

How to create repositories in Artifactory with curl

rndmh3ro — Mon, 30 Jun 2025 18:00:00 +0000

I recently created repositories in a new Artifactory instance. This was a testing instance and since I dind’t work with Artifactory much before this, I created them in the web-frontend by hand.

I then wanted to get them as code so I could recreate them in the production Artifactory instance without doing it all by hand.

Since I had admin-privileges in the testing instance, I used the /repositories/configurations-endpoint to get the repositories and their configuration in json-format. I then wanted to create these repositories on the production instance.

For this I built myself this shell one-liner.

curl -s -u <USER>:<PASSWORD> "https://artifactory-test.example.com/artifactory/api/repositories/configurations" | jq .REMOTE | curl --location --request PUT -u <USER>:<PASSWORD> "https://artifactory.example.com/artifactory/api/v2/repositories/batch" --header 'Content-Type: application/json' --data @-

It first reads all repositories from the testing instance, uses jq to get the remote-repositories, then PUT these into the production-instance, using the /repositories/batch-endpoint.

This would have worked, had I got admin-privileges on the production instance, too. But sadly I didn’t!

The /repositories/configurations-endpoint can only be used as an admin and there is no endpoint to get the configurations of all repositories you have access to.

But there is an endpoint to get all the repositories without the configurations. So all I had to do is get the names of the repositories (in the json-key .key) I had access to, iterate over them and then get the configuration of each repository.

for repo in $(curl -s -u <USER>:<PASSWORD> "https://artifactory.example.com/artifactory/api/repositories/?project=ocp" | jq -r .[].key); do curl -s -u <USER>:<PASSWORD> "https://artifactory.example.com/artifactory/api/repositories/$repo" | jq .; done | jq -s '.' > repos.json

With the repos.json-file containing the configuration of all the repositories I have access to, I can now create (by PUTing them) the repositories on the production instance:

curl -X PUT -s -u <USER>:<PASSWORD> 'https://artifactory.example.com/artifactory/api/v2/repositories/batch' --header 'Content-Type: application/json' --data @repos.json

If I want to update them, I can use almost the same command, but instead I need to POST them:

curl -X POST -s -u <USER>:<PASSWORD> 'https://artifactory.example.com/artifactory/api/v2/repositories/batch' --header 'Content-Type: application/json' --data @repos.json

Next step (?) - put them into an Ansible-module.

Nachbericht Stackconf 2025

rndmh3ro — Wed, 21 May 2025 08:15:00 +0000

Dieses Jahr war ich zum ersten Mal auf der Stackconf. Da Netways auch diese Konferenz organisiert, war es wie gewohnt schön, alte Bekannte und auch neue Gesichter zu treffen. Es ist stets interessant, in den Gesprächen mit Kollegen aus unterschiedlichen Bereichen, von großen als auch kleinen Unternehmen, national und international, über die verschiedenen Themen und Perspektiven auch abseits der Session Themen zu diskutieren.

Da noch nicht alle Videos veröffentlicht wurden, wollte ich dennoch mit wenigen Worten den Hauptinhalt der Sessions wiedergeben.

Tag 1

Los ging es mit dem Thema The Sustainable Infrastructure of the future.

Vereinfacht ging es in dem Vortrag um praktische Tipps zum Thema Green IT, die selbst von Personen die dem Thema eher skeptisch gegenüberstehen ein Gedanke wert sein können.

Ein paar Beispiele aus dem Vortrag hierfür waren:

Autoscaling, effiziente Nutzung der Ressourcen
Systeme runterfahren wenn Sie nicht gebraucht werden, z.B. am Wochenende
Cloud statt eigene Server - Ressourcen Auslastung
“Follow the Sun” - die Server laufen dort wo auch gerade Solarenergie erzeugt werden kann
Ist es notwendig, jedes Mal ChatGPT zu fragen, wenn Stackoverflow die Antwort auch liefert?

Aber auch die klare Aussage des Redners, dass die meisten CO2-Ausgleichs-Zertifikate nurGreen Washing sind.

Direkt im Anschluss ging es weiter mit Detect & Respond to Threats in Kubernetes with Falco. Hierbei handelte es sich um die Vorstellung des Open Source Security Scanners falco.

Interessant daran war, dass falco die Container zur Laufzeit prüft. Dabei kommt der Scanner mit einem vordefiniertem Ruleset, das aber angepasst werden kann sowie eigene Rule Definitionen ermöglicht.

Das Tool kann in der Cloud aber auch direkt auf dem Host genutzt werden. Es werden sogar direkt Helm-Charts und Docker-Container aber auch Softwarepakete bereitgestellt.

Beispiele für findings während der Demo waren:

Es gibt eine laufende binary, die nicht Teil des Images ist
Innerhalb des containers wurde eine shell Session gestartet

Da es sich hier um ein OpenSource Tool handelt, könnte es als Alternative zu den Cloud-Angeboten (z.B. Azure Defender) betrachtet werden. Sowohl im Hinblick auf Kosten als auch Vendor-Lock-In.

Nach diesen Sessions gab es erstmal einen kurzen Break und die erste Möglichkeit zum Austausch vor Ort.

Danach ging es weiter mit der eher an Entwickler gerichteten Session Integrating generative AI into API Platform: Good idea?. Hier ging es um eine Vorstellung was möglich ist in Bezug auf die Anbindung von KI-APIs.

Innerhalb des Vortrages gab es interessante zusätzliche Aussagen:

Es ist immer noch schwer für Unternehmen einen sinnvollen Anwendungsfall für KI zu finden.
Im Hinblick auf Green IT sollte man aber auch die Frage “Muss man die KI anbinden oder macht auch eine Alternative (z.B. Search Engine) mehr Sinn?” betrachten.

Darauf folgte How NVMe over TCP runs PostgreSQL in Quicksilver mode! und MultiCloud: Behind the Hype. Beim letzteren ging es inhaltlich und die Erläuterung welchen Formen von verschiedenen Cloud Typen es gibt und wieso weshalb warum man sich für die verschiedenen Typen entscheiden sollte. Kernaussage war das aktuell meist ein hybrider Ansatz erfolgt wird.

Nach der Mittagspause und den sich anschließenden Ignite-Talks ging es weiter mit den Themen Embracing the Local-First Paradigm und The Power of Small Habits in Agile Teams.

Nach einer Kaffeepause folgte ein längerer Open Spaces Block. Beim Open Spaces-Konzept können Konferenzteilnehmer Themen platzieren, über die sie sich mit anderen Teilnehmern austauschen möchten.

Zum Abschluss des Tages gab es noch eine, sagen wir Leidensgeschichte, unter dem Thema IP Authentication: A Tale of Performance Pitfalls and Challenges in Prod.

Hieran hat mir vor allem gefallen, dass es kein komplexes Thema und “wir haben das beim ersten Mal gerockt”-Vortrag war. Stattdessen wurde darauf eingegangen, wurde wie oft man den Livegang gemacht hat, wie der Rollback lief und was man dazwischen gemacht hat.

Damit war der erste Tag offiziell beendet und man ging in die Abendveranstaltung über. Für alle die schon auf Netways-Events waren, wissen wie gut aber auch langwierig diese sind :).

Tag 2

Der Tag startete mit der Session 2025: I Don’t Know K8S and at This Point, I’m Too Afraid To Ask. Wie der Titel es schon verrät, ging es um Kubernetes. Inhaltich war es einer schöner Crashkurs zum Thema. Wer einfach einen schnellen Überblick über das Thema haben möchte ohne direkt in eine 1-Tages-Schulung zu müssen, sollte sich diesen Vortrag ruhig mal ansehen.

Der folgende Vortrag Evolving Shift Left: Integrating Observability into Modern Software Development war eine Vorstellung zu Opentelemetry.

Nach einer Kaffeepause ging es weiter mit Best Practices zum Setup von APIs - Breaking APIs: how to cook up the perfect design. Im darauf folgenden Vortrag Building a Hyperconverged Proxmox VE Cluster with Ceph präsentierte der Redner wie Sie in ihrem Unternehmen mittels Ceph in ihrem Proxmox Cluster Storage bereitgestellt haben.

Nach der Mittagspause und den darauf folgenden Ignites ging es weiter mit How Open Source Communities are Defining the Next Generation of Infrastructure. Inhaltlich ging es hier um eine Vorstellung von OpenStack und Werbung Teil der Community zu werden.

Im Vortrag Zap the Flakes! Leveraging AI to Combat Flaky Tests with CANNIER wurde RedHat Cannier (von einem RedHat Mitarbeiter) vorgestellt und auf den aktuellen Implementierungsstand eingegangen.

Nach einer kleinen Pause gab es auch am 2. Tag wieder Open Spaces. Zum Abschluss folgte noch die Session Operator All the (stateful) Things. Hier würde auf die Verwaltung von Datenbanken as Code mit Atlas präsentiert.

Zusammenfassend kann ich sagen, dass es mal wieder ein rundum guter Konferenzbesuch mit vielen Eindrücken und Impulsen war.

Folgende 4 Vorträge möchte ich aus den folgenden Gründen hervorheben.

Hat mich positiv überrascht - The Sustainable Infrastructure of the future
Sollten wir ausprobieren - Detect & Respond to Threats in Kubernetes with Falco
Motivation / positives Gefühl steigern was man doch so alles schafft - The Power of Small Habits in Agile Teams
Schneller Überblick über das komplexe Thema - 2025: I Don’t Know K8S and at This Point, I’m Too Afraid To Ask

TIL how to test CORS on the command line with curl

rndmh3ro — Sat, 28 Sep 2024 19:30:00 +0000

In my last TIL I talked about how to set additional security headers for Gitlab. But I also had to do this for other applications I was supporting, where it was more straight-forward to do it (meaning: with code).

I needed to set the access-control-allow-origin header in the other applications. This header tells the browser from which origins the website is allowed to download resources.

This is relevant in web apps that run on webapp-frontend.example.com and want to use data from webapp-backend.example.com. To allow the browser to access webapp-backend.example.com, the frontend application needs to allow it in its access-control-allow-origin header, like this:

access-control-allow-origin: https://webapp-backend.example.com

So I set this in the frontend-application and wanted to test it from the command line.

Here’s how I did it:

curl -v --request OPTIONS 'https://webapp-frontend.example.com' -H 'Origin: https://webapp-backend.example.com' -H 'Access-Control-Request-Method: POST'

Of course I used curl for testing. To check if access-control-allow-origin works, you send a OPTIONS-request to check what methods are allowed by the application. The header 'Access-Control-Request-Method: POST' informs the application that I want to do a POST-request (I guess that’s not strictly necessary to test the allow-origin header).

Now to finally test it, I deactivated the access-control-allow-origin header, to see what will happen. Here’s the result:

> curl -v --request OPTIONS 'https://webapp-frontend.example.com' -H 'Origin: https://webapp-backend.example.com' -H 'Access-Control-Request-Method: POST'
* Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> OPTIONS /healthz HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.7.1
> Accept: */*
> Origin: https://webapp-backend.example.com
> Access-Control-Request-Method: POST
> 
* Request completely sent off
< HTTP/1.1 400 Bad Request
< date: Fri, 06 Sep 2024 11:44:27 GMT
< server: uvicorn
< vary: Origin
< access-control-allow-methods: GET, POST
< access-control-max-age: 600
< access-control-allow-headers: Accept, Accept-Language, Content-Language, Content-Type
< content-length: 22
< content-type: text/plain; charset=utf-8
< content-security-policy: base-uri 'self'; connect-src 'self'; script-src; script-src-attr; frame-src; object-src 'none'; img-src; style-src; font-src; manifest-src; media-src; default-src
< referrer-policy: no-referrer
< x-frame-options: DENY
< x-content-type-options: nosniff
< cross-origin-embedder-policy: require-corp
< cross-origin-opener-policy: same-origin
< cross-origin-resource-policy: same-origin
< 
* Connection #0 to host 127.0.0.1 left intact
Disallowed CORS origin

The last line tells you that my request was denied. You can also see that there are three access-control headers in the response (access-control-allow-methods, access-control-max-age, access-control-allow-headers), but no access-control-allow-origin.

Now I activate the header in the application and re-run the curl command:

> curl -v --request OPTIONS 'https://webapp-frontend.example.com' -H 'Origin: https://webapp-backend.example.com' -H 'Access-Control-Request-Method: POST'
* Trying 127.0.0.1:8000...
* Connected to 127.0.0.1 (127.0.0.1) port 8000
> OPTIONS /healthz HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/8.7.1
> Accept: */
> Origin: https://webapp-backend.example.com
> Access-Control-Request-Method: POST
> 
* Request completely sent off
< HTTP/1.1 200 OK
< date: Fri, 06 Sep 2024 11:44:07 GMT
< server: uvicorn
< vary: Origin
< access-control-allow-methods: GET, POST
< access-control-max-age: 600
< access-control-allow-headers: Accept, Accept-Language, Content-Language, Content-Type
< access-control-allow-origin: https://webapp-backend.example.com
< content-length: 2
< content-type: text/plain; charset=utf-8
< content-security-policy: base-uri 'self'; connect-src 'self'; script-src; script-src-attr; frame-src; object-src 'none'; img-src; style-src; font-src; manifest-src; media-src; default-src
< referrer-policy: no-referrer
< x-frame-options: DENY
< x-content-type-options: nosniff
< cross-origin-embedder-policy: require-corp
< cross-origin-opener-policy: same-origin
< cross-origin-resource-policy: same-origin
< 
* Connection #0 to host 127.0.0.1 left intact
OK

Now two things have changed:

I get back a OK-response indicating that my request ist allowed.
There’s now a header in the response: access-control-allow-origin: https://webapp-backend.example.com

And that’s it. With this simple method you can test on the command-line if your access-control-allow-origin works.

TIL how to configure additional headers in Gitlab’s nginx

rndmh3ro — Tue, 24 Sep 2024 13:30:00 +0000

Recently, I had to configure some security headers in GitLab. GitLab uses Nginx as its web server, and it allows for easy configuration changes for some settings. For instance, enabling HTTP to HTTPS redirection can be done simply by setting nginx['redirect_http_to_https'] = true in the gitlab.rb configuration file.

However, adding custom headers for security, particularly those that control cross-origin policies, requires a bit more work. These headers are essential for preventing certain types of attacks and ensuring better isolation between websites.

I needed to set three headers: Cross-Origin-Opener-Policy (COOP), Cross-Origin-Embedder-Policy (COEP), and Cross-Origin-Resource-Policy (CORP). These headers are used to prevent cross-origin attacks, such as Spectre, and ensure that only resources from trusted origins can interact with the site.

COOP ensures that the window or tab in which the site is running is isolated from any other cross-origin content. COEP guarantees that cross-origin resources can only be embedded if they explicitly grant permission. CORP restricts which origins can access certain resources, preventing untrusted external sites from accessing sensitive content.

Configuring these in Gitlab’s Nginx was a bit tricky. Nginx requires that every setting ends with a semicolon and a newline. Additionally, for better readability in the gitlab.rb file, I added line breaks while ensuring there were no spaces after the backslashes at the end of each line. Here’s the final configuration:

Here’s the final result that you can copy-paste into your gitlab.rb configuration file:

nginx['custom_gitlab_server_config'] = "add_header Cross-Origin-Opener-Policy same-origin;\n\
                                        add_header Cross-Origin-Embedder-Policy require-corp;\n\
                                        add_header Cross-Origin-Resource-Policy same-site;"

The key thing to note is that there should be no trailing spaces after the backslashes in this multi-line string, as even a single space could cause the configuration to fail.

Of course you can then add additional headers or other nginx-settings after the add_header-settings.

TIL how to define different Helm-Repos in a template

rndmh3ro — Wed, 04 Sep 2024 09:30:00 +0000

Recently I had to create a Helm-Chart (still not a fan of it, at all!) where the image was different depending on if the helm-chart was used for local development or used in production.

I had to resort to using an if-else-condition that I put into the _helpers.tpl-file so it can be accessed by any deployment-template in the chart.

{{- /*
Set registry to local docker-registy if deploying locally, else use prodution-registry.
If you use := in if-statement, it will declare a new local variable $env which will not impact the value of $env declared outside the if block.
*/}}

{{- define "ImageFunctions" -}}
{{- $Image := "" }}

{{- if (default .root.Values.LOCAL_DEV false) -}}
    {{- $Image = printf "%s:latest" $name | quote -}}
{{- else -}}
    {{- $Image = printf "%s/%s/%s:latest" "image-registry.openshift-image-registry.svc:5000" .root.Release.Namespace $name | quote -}}
{{- end -}}

{{- $Image -}}
{{- end }}

Let’s break it down:

First I define a function called “ImageFunctions” that will be then called in the deployment-templates. This functions first defines an empty Image-variable. This is needed, because if you define the variable only inside the following if-statement, it won’t be accessible outside the statement and we then cannot use it in our deployments.

Then comes the if-else condition. Here I use a Value LOCAL_DEV that is defined in a values-file that will only be used when developing locally. Otherwise it is empty and defaults to false. This way I only have to define this variable once for local development and not have to define it in the values-files for the production environment.

Inside the if function I set the Image-variable to “%s:latest” using a printf-statement. When templating this, it will resolve to for example “frontend-app:latest”. The else-condition does the same, except it prints a different image-name that can be used in production.

The line {{- $Image -}} then propagates the variable to outside the ImageFunctions-functions so it then can be used in the template.

Here’s how to use it in your deployment-template:

{{- $Image := include "ImageFunctions" $dict -}}
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: container
          image: {{ $Image }}

Let’s break this down, too:

The first line defines a new variable Image that used the defined ImageFunctions. It will be filled the the Image-variable coming from the function and will contain either frontend-app:latest or image-registry.openshift-image-registry.svc:5000:/frotend-app:latest, depending on if LOCAL_DEV was set to true or false.

Now you can just insert the first line into every of your deployment-templates and then use the $Image-variable. Then the correct image will be used, no matter if you use the chart locally or in production.

Working with Gitlab on the CLI

rndmh3ro — Mon, 03 Jun 2024 13:00:00 +0000

Glab is an open-source tool that allows you to work with GitLab from the command line, eliminating the need to switch to a browser to create or approve merge requests, start a pipeline run, or view issues.

Glab can work with repositories hosted on gitlab.com as well as with your own GitLab instances. The tool automatically detects which instance it should work with.

The CLI tool was started by Clement Sam and has been an official GitLab product since 2022.

Setup

Glab can be installed in various ways. Since it is written in Golang, the executable can be easily downloaded and run from the releases page. Alternatively, Glab is also available in various package repositories. It runs on Linux, Windows, and macOS.

All installation options can be found here!

Registering with the GitLab Instance

Before working with repositories, you need to authenticate with the GitLab instance. For this, you need a Personal Access Token, which you can create in your GitLab profile. For the gitlab.com instance, you can create it here. Assign a name to the token and select the “api” and “write_repository” permissions. The generated token will be needed in the next step.

Now, log in to the GitLab instance using the token by running glab auth login and answering the prompts.

> glab auth login
? What GitLab instance do you want to log into? gitlab.com
- Logging into gitlab.com
? How would you like to login? Token

Tip: you can generate a Personal Access Token here https://gitlab.com/-/profile/personal_access_tokens
The minimum required scopes are 'api' and 'write_repository'.
? Paste your authentication token: **************************? Choose default git protocol HTTPS
? Authenticate Git with your GitLab credentials? Yes
- glab config set -h gitlab.com git_protocol https
✓ Configured git protocol
- glab config set -h gitlab.com api_protocol https
✓ Configured API protocol
✓ Logged in as rndmh3ro

You can verify a successful login with glab auth status.

> glab auth status
gitlab.com
  ✓ Logged in to gitlab.com as rndmh3ro (/home/segu/.config/glab-cli/config.yml)
  ✓ Git operations for gitlab.com configured to use https protocol.
  ✓ API calls for gitlab.com are made over https protocol
  ✓ REST API Endpoint: https://gitlab.com/api/v4/
  ✓ GraphQL Endpoint: https://gitlab.com/api/graphql/
  ✓ Token: **************************
git.example.com
  ✓ Logged in to git.example.com as segu (/home/segu/.config/glab-cli/config.yml)
  ✓ Git operations for git.example.com configured to use https protocol.
  ✓ API calls for git.example.com are made over https protocol
  ✓ REST API Endpoint: https://git.example.com/api/v4/
  ✓ GraphQL Endpoint: https://git.example.com/api/graphql/
  ✓ Token: **************************

Working with Repositories

Once successfully logged into the GitLab instance, you can work with repositories using glab.

Cloning a Repository

To clone repositories with glab, run glab repo clone path/to/repo, followed by an optional target directory.

> glab repo clone gitlab-org/cli
Cloning into 'cli'...
remote: Enumerating objects: 18691, done.
remote: Counting objects: 100% (72/72), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 18691 (delta 53), reused 39 (delta 37), pack-reused 18619
Receiving objects: 100% (18691/18691), 22.98 MiB | 5.97 MiB/s, done.
Resolving deltas: 100% (12391/12391), done.

If you have multiple repositories in a group to clone, you can do this using glab as well. Use the --group option (or -g) to clone all repositories in the group sequentially:

> GITLAB_HOST=gitlab.com glab repo clone -g gitlab-org
fatal: destination path 'verify-mr-123640-security-policy-project' already exists and is not an empty directory.
Cloning into 'verify-mr-123640'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
Cloning into 'without-srp'...
remote: Enumerating objects: 30, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 30 (delta 7), reused 4 (delta 4), pack-reused 16
Receiving objects: 100% (30/30), 5.94 KiB | 5.94 MiB/s, done.
Resolving deltas: 100% (9/9), done.
Cloning into 'container-scanning-with-sbom'...

Working with Merge Requests

After checking out the repository, you can start working on issues or merge requests (MRs).

A typical code change process often looks like this: You make your code changes, commit them, and push them to GitLab. Then, you want to create a merge request. Normally, you would now switch to the GitLab website to create the MR. Thanks to glab, you don’t need to leave the command line.

Using glab mr create, you can interactively create an MR. You will be guided through the creation process, where you specify the title and description, and then you will be asked if you want to create the MR directly or view it in the web frontend before creating it.

> glab mr create
? Choose a template Open a blank merge request
? Title: New Feature
? Description <Received>
? What's next? Submit

Creating merge request for test into master in gitlab-org/cli

!351 New Feature (test)
https://gitlab.com/gitlab-org/cli/-/merge_requests/1297

You can then view it. If you want to do this in the browser, run glab mr view with the --web (or -w) parameter.

> glab mr view -w 1297

You can also view the same content directly on the command line (including comments):

> glab mr view 1297 -R gitlab-org/cli
open • opened by rndmh3ro about 1 hour ago
docs: add installation options with wakemeops !1297

  ## Description

  Add installation options with wakemeops-repository.

  Note: I'm not affiliated with WakeMeOps, just a happy user.

  ## Related Issues

  Resolves #1363

  ## How has this been tested?

  not at all.

  ## Screenshots (if appropriate):

  ### Types of changes

  [] Bug fix (non-breaking change which fixes an issue)
  [] New feature (non-breaking change which adds functionality)
  [] Breaking change (fix or feature that would cause existing functionality
  to change)
  [✓] Documentation
  [] Chore (Related to CI or Packaging to platforms)
  [] Test gap

0 upvotes • 0 downvotes • 5 comments
Labels: Community contribution, documentation, linked-issue, tw::triaged, workflow::ready for review
Assignees: rndmh3ro
Reviewers: aqualls
Pipeline Status: success (View pipeline with `glab ci view add_wakemeops_docs`)
Approvals Status:
Rule "All Members" insufficient approvals (0/1 required):

Rule "/docs/" sufficient approvals (0/0 required):
Amy Qualls aqualls -

✓ This merge request has 1 changes

View this merge request on GitLab: https://gitlab.com/gitlab-org/cli/-/merge_requests/1297

If you’re not working on the codebase yourself but are reviewing MRs from others, you can list open merge requests with glab mr list.

> glab mr list
Showing 22 open merge requests on gitlab-org/cli (Page 1)

!1297 gitlab-org/cli!1297 docs: add installation options with wakemeops (main) ← (add_wakemeops_docs)
!1296 gitlab-org/cli!1296 fix(repo view): consider current host when viewing different repositories (#1362) (main) ← (1362_repo_view)
!1295 gitlab-org/cli!1295 fix: `glab mr delete` should work properly for forks (main) ← (fix_mr_delete)

Afterwards, you can check out the MR you want to review:

> glab mr checkout 1297

If you’re satisfied with the content, you can add a note to the Merge Request and then approve it directly:

> glab mr note -R gitlab-org/cli -m "LGTM"

> glab mr approve
- Approving Merge Request !1297
✓ Approved

And of course, you can also merge the MR right away:

> glab mr merge
? What merge method would you like to use? Rebase and merge
? What's next? Submit
✓ Rebase successful
! No pipeline running on test
✓ Rebased and merged
!1297 New Feature (test)
https://gitlab.com/gitlab-org/cli/-/merge_requests/1297

At the end of the day, you can view your merged MRs by using glab mr list options to see only merged (-M) or your own (-a @me) MRs:

> glab mr list -M -a @me
Showing 4 merged merge requests on gitlab-org/cli (Page 1)

!1279 gitlab-org/cli!1279 feat(schedule): Add commands to create and delete schedules (main) ← (create_del_sched)
!1176 gitlab-org/cli!1176 feat(schedule): Add command to run schedules (main) ← (run_schedule)
!1143 gitlab-org/cli!1143 docs: remove duplicate defaults in help (main) ← (fix_help_doc)
!1112 gitlab-org/cli!1112 feat(schedule): Add command to list schedules (main) ← (sched_list)

Working with Pipelines

Code changes are tested through an automatic CICD pipeline. Naturally, glab offers the ability to work with pipelines.

To start a pipeline on the main branch, use the following command:

> glab ci run -b main
Created pipeline (id: 540823), status: created, ref: main, weburl: https://git.example.com/example/project/-/pipelines/540823

You can view the status of the pipeline like this:

> glab ci status
(failed) • 01m 11s lint test

https://git.example.com/example/project/-/pipelines/540812
SHA: 275cb8295c69db166e1b1c94936d4c4b67463701
Pipeline State: failed

? Choose an action: Exit

If a pipeline has failed, you can view the logs using glab ci trace:

> glab ci trace

Searching for latest pipeline on test...
Getting jobs for pipeline 540812...

? Select pipeline job to trace: kics-scan (1209237) - failed

Getting job trace...
Showing logs for kics-scan job #1209237
Running with gitlab-runner 14.10.1 (f761588f)
  on example-shared-docker swAou6b9
Resolving secrets
Preparing the "docker" executor

glab works excellently with Unix pipes, so you can easily grep for errors:

> glab ci trace 1209237 | grep -i failed
Queries failed to execute: 10
ERROR: Job failed: exit code 50

Linting

Speaking of errors - you can incorporate them wonderfully into the CICD configuration file, the .gitlab-ci.yml.

If you change this file, for example to add a new stage, and make a mistake, you will usually only notice it after you have pushed your changes and wonder why the pipeline does not start.

Fortunately, you can check (“lint”) the configuration with glab.

If an error has crept in, glab ci lint will detect it.

> glab ci lint
Validating...
.gitlab-ci.yml is invalid
1 (<unknown>): did not find expected key while parsing a block mapping at line 2 column 1

After correction, the linting will then report success:

> glab ci lint
Validating...
✓ CI/CD YAML is valid!

Working with Schedules

I am particularly proud of the feature to create and run Pipeline Schedules with glab, because I implemented it.

Pipeline Schedules are designed to automatically run pipelines at regular intervals.

You can create these with glab. To do this, you pass a cron expression (which defines when the pipeline should run), a description, and the branch on which the pipeline should run:

> glab schedule create --cron "0 2 * * *" --description "Run main pipeline everyday" --ref "main" --variable "foo:bar"
Created schedule

You can view the created pipeline schedule with glab schedule list:

> glab schedule list
Showing 1 schedules on example/project (Page 1)

ID Description Cron Owner Active
1038 Run main pipeline everyday * * * * * segu true

To run the pipeline schedule outside the defined rhythm, start it with glab schedule run:

> glab schedule run 1038
Started schedule with ID 1038

And if it is no longer needed, you can simply delete it:

> glab schedule delete 1038
Deleted schedule with ID 1038

glab API

Not all functions that Gitlab offers are yet usable with glab. For such cases, it is possible to communicate directly with the Gitlab API using glab api.

The command to display the pipeline schedules (glab schedule list) mentioned in the previous section can be replicated using a glab api call:

> glab api projects/:fullpath/pipeline_schedules/
[
  {
    "id": 1038,
    "description": "Run main pipeline everyday",
    "ref": "main",
    "cron": "* * * * *",
    "cron_timezone": "UTC",
    "next_run_at": "2023-06-22T08:33:00.000Z",
    "active": true,
    "created_at": "2023-06-22T08:24:02.199Z",
    "updated_at": "2023-06-22T08:24:02.199Z",
    "owner": {
      "id": 97,
      "username": "segu",
      "name": "Sebastian Gumprich",
      "state": "active",
    }
  }
]

You can also delete the pipeline schedule this way:

> glab api projects/:fullpath/pipeline_schedules/1038 -X DELETE

> glab api projects/:fullpath/pipeline_schedules/1038
glab: 404 Pipeline Schedule Not Found (HTTP 404)
{
  "message": "404 Pipeline Schedule Not Found"
}

Aliases

To avoid having to remember the sometimes more complicated API calls, glab has functionality to create aliases.

Two aliases are already set up by default, which you can display as follows:

> glab alias list
ci pipeline ci
co mr checkout

So if you want to check out a merge request, you can simply call glab co instead of glab mr checkout.

You can define your own aliases as follows:

> glab alias set schedule_list 'api projects/:fullpath/pipeline_schedules/'
- Adding alias for schedule_list: api projects/:fullpath/pipeline_schedules/
✓ Added alias.

And of course, you can delete them again:

> glab alias delete schedule_list
✓ Deleted alias schedule_list; was api projects/:fullpath/pipeline_schedules/

Set Variables from GitlabCI Locally

Another useful glab feature is working with CICD variables.

You can display, create, and delete these as well:

> glab variable list

> glab variable set foo bar
✓ Created variable foo for 7001-07/nrwsp with scope *

> glab variable get foo
bar

> glab variable delete foo
✓ Deleted variable foo with scope * for 7001-07/nrwsp

The variables created this way can be used locally in a simple manner.

If you use Terraform, for example, you can set your TF_VAR variables easily by setting the output of glab variable get as an environment variable.

export TF_VAR_db_root_password=$(glab variable get TF_VAR_db_root_password)
export TF_VAR_secret_key=$(glab variable get TF_VAR_secret_key)
export TF_VAR_access_key=$(glab variable get TF_VAR_access_key)

If you copy these exports into your README, each team member can set the correct Terraform variables with a simple copy-paste, without having to copy them from a password manager in a cumbersome way.

Bash-Completion and Further Information

If you want to know what else glab can do - the bash autocompletion shows it to you:

And many more details can of course be found on the Homepage of glab.

Gitlab von der Kommandozeile aus bedienen

rndmh3ro — Mon, 03 Jun 2024 13:00:00 +0000

Glab ist ein Opensource-Tool, das es ermöglicht mit Gitlab über die Kommandozeile zu arbeiten. Dadurch entfällt das Wechseln zum Browser, um Merge Requests zu erstellen oder zu genehmigen, einen Pipeline-Lauf zu starten oder Issues anzusehen.

Glab kann mit Repositories arbeiten, die auf gitlab.com gehostet sind, aber auch mit eigenen Gitlab-Instanzen. Das Tool erkennt automatisch, mit welcher Instanz es gerade arbeiten soll.

Das CLI-Tool wurde von Clement Sam gestartet und ist seit 2022 ein offizielles Gitlab-Produkt.

Setup

Glab kann auf verschiedene Arten installiert werden. Da es in Golang geschrieben ist, kann die ausführbare Datei problemlos über die Releases-Seite heruntergeladen und ausgeführt werden. Alternativ ist Glab auch in verschiedenen Paket-Repositories verfügbar. Es läuft unter Linux, Windows und macOS.

Alle Installationsvarianten kann man hier finden!

Registrierung an der GitLab-Instanz

Bevor man mit Repositories arbeiten kann, muss man sich an der Gitlab-Instanz authentifizieren. Hierfür benötigt man ein Personal Access Token, welches man sich in seinem Gitlab-Profil erstellen kann. Im Falle der gitlab.com-Instanz ist die Erstellung hier zu finden. Man vergibt einen beliebigen Namen für das Token und wählt die Berechtigungen “api” und “write_repository”. Das generierte Token wird dann im nächsten Schritt benötigt.

Nun meldet man sich mit dem Token an der Gitlab-Instanz an, indem man glab auth login ausführt und die gestellten Fragen beantwortet.

> glab auth login
? What GitLab instance do you want to log into? gitlab.com
- Logging into gitlab.com
? How would you like to login? Token

Tip: you can generate a Personal Access Token here https://gitlab.com/-/profile/personal_access_tokens
The minimum required scopes are 'api' and 'write_repository'.
? Paste your authentication token: **************************? Choose default git protocol HTTPS
? Authenticate Git with your GitLab credentials? Yes
- glab config set -h gitlab.com git_protocol https
✓ Configured git protocol
- glab config set -h gitlab.com api_protocol https
✓ Configured API protocol
✓ Logged in as rndmh3ro

Den erfolgreichen Login kann man mittels glab auth status überprüfen.

> glab auth status
gitlab.com
  ✓ Logged in to gitlab.com as rndmh3ro (/home/segu/.config/glab-cli/config.yml)
  ✓ Git operations for gitlab.com configured to use https protocol.
  ✓ API calls for gitlab.com are made over https protocol
  ✓ REST API Endpoint: https://gitlab.com/api/v4/
  ✓ GraphQL Endpoint: https://gitlab.com/api/graphql/
  ✓ Token: **************************
git.example.com
  ✓ Logged in to git.example.com as segu (/home/segu/.config/glab-cli/config.yml)
  ✓ Git operations for git.example.com configured to use https protocol.
  ✓ API calls for git.example.com are made over https protocol
  ✓ REST API Endpoint: https://git.example.com/api/v4/
  ✓ GraphQL Endpoint: https://git.example.com/api/graphql/
  ✓ Token: **************************

Mit Repositories arbeiten

Hat man sich erfolgreich an der GitLab-Instanz angemeldet, kann man mittels glab mit Repositories arbeiten.

Repository klonen

Zu erst einmal will man natürlich Repositories mittels glab clonen. Dazu ruft man glab repo clone path/to/repo auf, gefolgt von einem optionalen Zielverzeichnis.

> glab repo clone gitlab-org/cli
Cloning into 'cli'...
remote: Enumerating objects: 18691, done.
remote: Counting objects: 100% (72/72), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 18691 (delta 53), reused 39 (delta 37), pack-reused 18619
Receiving objects: 100% (18691/18691), 22.98 MiB | 5.97 MiB/s, done.
Resolving deltas: 100% (12391/12391), done.

Hat man mehrere Repositories in einer Gruppe, die man clonen möchte, kann man dies mittels glab ebenfalls tun. Hierzu muss man die --group-Option (oder -g) nutzen. Damit werden nacheinander alle Repositories der Gruppe gecloned:

> GITLAB_HOST=gitlab.com glab repo clone -g gitlab-org
fatal: destination path 'verify-mr-123640-security-policy-project' already exists and is not an empty directory.
Cloning into 'verify-mr-123640'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
Cloning into 'without-srp'...
remote: Enumerating objects: 30, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 30 (delta 7), reused 4 (delta 4), pack-reused 16
Receiving objects: 100% (30/30), 5.94 KiB | 5.94 MiB/s, done.
Resolving deltas: 100% (9/9), done.
Cloning into 'container-scanning-with-sbom'...

Mit Merge Requests arbeiten

Nachdem das Repository ausgecheckt wurde, kann man mit der Arbeit an Issues oder Merge Requests (MRs) beginnen.

Ein normaler Prozess zur Codeänderung sieht meist so aus: Man führt seine Codeänderungen durch, committed sie und pushed sie anschließend in das Gitlab. Daraufhin möchte man einen Merge Request erstellen. Normalerweise würde man nun zur Gitlab-Website wechseln und dort den MR erstellen. Dank glab muss man die Kommandozeile aber nicht verlassen.

Mittels glab mr create wird interaktiv ein MR erstellt. Dabei wird man durch den Erstellungsprozess geführt, indem man Titel und Beschreibung angibt und anschließend gefragt wird, ob man den MR direkt erstellen will oder ihn sich im Webfrontend ansehen will, bevor er erstellt wird.

> glab mr create
? Choose a template Open a blank merge request
? Title: New Feature
? Description <Received>
? What's next? Submit

Creating merge request for test into master in gitlab-org/cli

!351 New Feature (test)
https://gitlab.com/gitlab-org/cli/-/merge_requests/1297

Danach kann man ihn sich ansehen. Möchte man das im Browser tun, ruft man glab mr view mit dem Parameter --web (oder -w) auf.

> glab mr view -w 1297

Man kann sich die gleichen Inhalte aber auch direkt auf der Kommandozeile ansehen (inklusive Kommentare):

> glab mr view 1297 -R gitlab-org/cli
open • opened by rndmh3ro about 1 hour ago
docs: add installation options with wakemeops !1297

  ## Description

  Add installation options with wakemeops-repository.

  Note: I'm not affiliated with WakeMeOps, just a happy user.

  ## Related Issues

  Resolves #1363

  ## How has this been tested?

  not at all.

  ## Screenshots (if appropriate):

  ### Types of changes

  [] Bug fix (non-breaking change which fixes an issue)
  [] New feature (non-breaking change which adds functionality)
  [] Breaking change (fix or feature that would cause existing functionality
  to change)
  [✓] Documentation
  [] Chore (Related to CI or Packaging to platforms)
  [] Test gap

0 upvotes • 0 downvotes • 5 comments
Labels: Community contribution, documentation, linked-issue, tw::triaged, workflow::ready for review
Assignees: rndmh3ro
Reviewers: aqualls
Pipeline Status: success (View pipeline with `glab ci view add_wakemeops_docs`)
Approvals Status:
Rule "All Members" insufficient approvals (0/1 required):

Rule "/docs/" sufficient approvals (0/0 required):
Amy Qualls aqualls -

✓ This merge request has 1 changes

View this merge request on GitLab: https://gitlab.com/gitlab-org/cli/-/merge_requests/1297

Arbeitet man gerade nicht selbst an der Codebasis sondern sieht sich MRs anderer Personen an, kann man sich mittels glab mr list offene Merge-Requests anzeigen lassen.

> glab mr list
Showing 22 open merge requests on gitlab-org/cli (Page 1)

!1297 gitlab-org/cli!1297 docs: add installation options with wakemeops (main) ← (add_wakemeops_docs)
!1296 gitlab-org/cli!1296 fix(repo view): consider current host when viewing a repo details (main) ← (jmc-1334)
!1295 gitlab-org/cli!1295 chore(ci): remove ssh key from build (main) ← (jmc-remove-ssh)

Im Anschluss checkt man den MR aus, den man sich ansehen möchte:

> glab mr checkout 1297

Ist man mit dem Inhalt zufrieden, kann man dem Merge Request zum Beispiel eine Notiz hinzufügen und anschließend direkt approven:

> glab mr note -R gitlab-org/cli -m "LGTM"

> glab mr approve
- Approving Merge Request !1297
✓ Approved

Und natürlich kann man den MR auch gleich mergen:

> glab mr merge
? What merge method would you like to use? Rebase and merge
? What's next? Submit
✓ Rebase successful
! No pipeline running on test
✓ Rebased and merged
!1297 New Feature (test)
https://gitlab.com/gitlab-org/cli/-/merge_requests/1297

Um sich am Ende des Tages seine gemergeden MRs anzusehen, gibt man glab mr list Optionen mit, um nur gemergede (-M) oder meine eigenen (-a @me) MRs anzusehen:

> glab mr list -M -a @me
Showing 4 merged merge requests on gitlab-org/cli (Page 1)

!1279 gitlab-org/cli!1279 feat(schedule): Add commands to create and delete schedules (main) ← (create_del_sched)
!1176 gitlab-org/cli!1176 feat(schedule): Add command to run schedules (main) ← (run_schedule)
!1143 gitlab-org/cli!1143 docs: remove duplicate defaults in help (main) ← (fix_help_doc)
!1112 gitlab-org/cli!1112 feat(schedule): Add command to list schedules (main) ← (sched_list)

Mit Pipelines arbeiten

Code-Änderungen werden durch eine automatische CICD-Pipeline getestet. Natürlich bietet glab die Möglichkeit, mit Pipelines zu arbeiten.

Zum Starten einer Pipeline auf dem Main-Branch ruft man folgenden Befehl auf:

> glab ci run -b main
Created pipeline (id: 540823 ), status: created , ref: main , weburl: https://git.example.com/example/project/-/pipelines/540823 )

Den Status der Pipeline kann man sich so ansehen:

> glab ci status
(failed) • 01m 11s lint test

https://git.example.com/example/project/-/pipelines/540812
SHA: 275cb8295c69db166e1b1c94936d4c4b67463701
Pipeline State: failed

? Choose an action: Exit

Ist eine Pipeline fehlgeschlagen, kann man sich die Logs mittels glab ci trace ansehen:

> glab ci trace

Searching for latest pipeline on test...
Getting jobs for pipeline 540812...

? Select pipeline job to trace: kics-scan (1209237) - failed

Getting job trace...
Showing logs for kics-scan job #1209237
Running with gitlab-runner 14.10.1 (f761588f)
  on example-shared-docker swAou6b9
Resolving secrets
Preparing the "docker" executor

glab arbeitet hervorragend mit Unix-Pipes zusammen - so kann man sich Fehler einfach heraus greppen:

> glab ci trace 1209237 | grep -i failed
Queries failed to execute: 10
ERROR: Job failed: exit code 50

Linting

Wo wir von Fehlern sprechen - man kann sie wunderbar in der CICD-Konfigurationsdatei, der .gitlab-ci.yml einbauen.

Ändert man diese Datei, weil man zum Beispiel eine neue Stage einbauen möchte und macht dabei einen Fehler, bekommt man diesen normalerweise erst mit, wenn man seine Änderungen gepushed hat und sich wundert, warum die Pipeline nicht losläuft.

Zum Glück kann man mit glab die Konfiguration überprüfen (“linten”).

Hat sich ein Fehler eingeschlichen, wird glab ci lint diesen feststellen.

> glab ci lint
Validating...
.gitlab-ci.yml is invalid
1 (<unknown>): did not find expected key while parsing a block mapping at line 2 column 1

Nach der Korrektur meldet das Linting dann Erfolg:

> glab ci lint
Validating...
✓ CI/CD YAML is valid!

Mit Schedules arbeiten

Auf das Feature, Pipeline Schedules mittels glab zu erstellen und laufen zu lassen, bin ich ganz besonders stolz, denn ich habe sie implementiert.

Pipeline Schedules sind dafür da, Pipelines in regelmäßigen Abständen automatisch laufen zu lassen.

Man kann diese per glab anlegen lassen. Dazu übergibt man dem Befehl eine Cron-Expression (die definiert, wann die Pipeline laufen soll), eine Beschreibung sowie den Branch, auf dem die Pipeline laufen soll:

> glab schedule create --cron "0 2 * * *" --description "Run main pipeline everyday" --ref "main" --variable "foo:bar"
Created schedule

Die so angelegte Pipeline schedule kann man sich per glab schedule list ansehen:

> glab schedule list
Showing 1 schedules on example/project (Page 1)

ID Description Cron Owner Active
1038 Run main pipeline everyday * * * * * segu true

Um die Pipeline Schedule außerhalb des definierten Rythmus laufen zu lassen, startet man sie mit glab schedule run:

> glab schedule run 1038
Started schedule with ID 1038

Und wird sie nicht mehr benötigt, kann man sie einfach löschen:

> glab schedule delete 1038
Deleted schedule with ID 1038

glab API

Noch sind nicht alle Funktionen, die Gitlab bietet, auch mittels glab nutzbar. Für solche Fälle ist es möglich, mittels glab api direkt mit der Gitlab-API zu kommunizieren.

Den im vorigen Abschnitt erwähnte Befehl zum Anzeigen der Pipeline Schedules (glab schedule list) kann man mittels eines glab api-Aufrufes nachbilden:

> glab api projects/:fullpath/pipeline_schedules/
[
  {
    "id": 1038,
    "description": "Run main pipeline everyday",
    "ref": "main",
    "cron": "* * * * *",
    "cron_timezone": "UTC",
    "next_run_at": "2023-06-22T08:33:00.000Z",
    "active": true,
    "created_at": "2023-06-22T08:24:02.199Z",
    "updated_at": "2023-06-22T08:24:02.199Z",
    "owner": {
      "id": 97,
      "username": "segu",
      "name": "Sebastian Gumprich",
      "state": "active",
    }
  }
]

Auch das Löschen der Pipeline Schedule ist so möglich:

> glab api projects/:fullpath/pipeline_schedules/1038 -X DELETE

> glab api projects/:fullpath/pipeline_schedules/1038
glab: 404 Pipeline Schedule Not Found (HTTP 404)
{
  "message": "404 Pipeline Schedule Not Found"
}

Aliases

Damit man sich die teils komplizierteren API-Aufrufe nicht merken muss, existiert in glab die Funktionalität, Aliase anzulegen.

Es sind bereits zwei Aliase standardmäßig eingerichtet, die man sich wie folgt anzeigen lassen kann:

> glab alias list
ci pipeline ci
co mr checkout

Möchte man also einen Merge Request auschecken, kann man statt glab mr checkout einfach glab co aufrufen.

Eigene Aliase kann man wie folgt definieren:

> glab alias set schedule_list 'api projects/:fullpath/pipeline_schedules/'
- Adding alias for schedule_list: api projects/:fullpath/pipeline_schedules/
✓ Added alias.

Und natürlich kann man sie auch wieder löschen:

> glab alias delete schedule_list
✓ Deleted alias schedule_list; was api projects/:fullpath/pipeline_schedules/

Variablen aus der GitlabCI lokal setzen

Ein weiteres nützliches glab-Feature ist, mit CICD-Variablen zu arbeiten.

Auch diese kann man sich anzeigen lassen, sie erstellen und sie löschen:

> glab variable list

> glab variable set foo bar
✓ Created variable foo for example/project with scope *

> glab variable get foo
bar

> glab variable delete foo
✓ Deleted variable foo with scope * for example/project

Die so erstellten Variablen lassen sich auf einfache Art und Weise auch lokal nutzen.

Nutzt man beispielsweise Terraform, kann man seine TF_VAR-Variablen ganz einfach setzen, indem man die Ausgabe von glab variable get als Umgebungsvariable setzt.

export TF_VAR_db_root_password=$(glab variable get TF_VAR_db_root_password)
export TF_VAR_secret_key=$(glab variable get TF_VAR_secret_key)
export TF_VAR_access_key=$(glab variable get TF_VAR_access_key)

Kopiert man diese exports in seine README, kann jedes Teammitglieder mit einem einfachen Copy-Paste die korrekten Terraform-Variablen setzen, ohne sie umständlich aus einem Passwort-Manager zu kopieren.

Bash-Completion und weitere Infomationen

Wer nun wissen möchte, was glab noch alles kann - die bash autocompletion zeigt es einem:

Und viele weitere Informationen findet man natürlich auf der Homepage von glab.

Interesting Uses of Ansible’s ternary filter

rndmh3ro — Wed, 21 Feb 2024 20:00:00 +0000

Some time ago I discovered an interesting use of the ternary-filter in Ansible. A ternary-filter in Ansible is a filter that takes three arguments: a condition, an value if the condition is true, and an alternative value if the condition is false.

Here’s a simple example straight from Ansible’s documentation:

- name: service-foo, use systemd module unless upstart is present, then use old service module
  service:
    state: restarted
    enabled: yes
    use: "{{ (ansible_service_mgr == 'upstart') | ternary('service', 'systemd') }}"

But there are many more interesting use cases for this filter and I decided to take a look what Ansible’s collection authors used it for.

Display command output only if verbosity is greater than 0.

This was the usage that initially got me interested in different use-cases for the filter.

Depending on what verbosity level you use when running a playbook (e.g. how many -v you add to the command), the command will run in quiet-mode (with the -quiet flag) or not.

- name: Validate configuration
  become: true
  become_user: "{{ consul_user }}"
  ansible.builtin.command: >
    {{ consul_binary }} validate {{ (ansible_verbosity == 0) | ternary("-quiet", "") }}
    {{ consul_config_path }}/config.json {{ consul_configd_path }}
  changed_when: false

(Source)

Testing task idempotency in one run without molecule

This use of the ternary-filter is useful for testing task idempotency in one run.

First, the task-file test_create_scheduler.yml is imported without a variable set, so the task will change something. Then, the task-file is imported again, however this time with the variable test_proxysql_scheduler_check_idempotence set to true.

- name: "{{ role_name }} | test_create_scheduler | test create scheduler"
  import_tasks: test_create_scheduler.yml

- name: "{{ role_name }} | test_create_scheduler | test idempotence of create scheduler"
  import_tasks: test_create_scheduler.yml
  vars:
    test_proxysql_scheduler_check_idempotence: true

When importing the test file test_create_scheduler.yml without the variable test_proxysql_scheduler_check_idempotence, the assert will check for status is changed, because the ternary-filter evaluated the variable test_proxysql_scheduler_check_idempotence as false.

- name: "{{ role_name }} | {{ current_test }} | check if create scheduler reported a change"
  assert:
    that:
      - "status is {{ test_proxysql_scheduler_check_idempotence|ternary('not changed', 'changed') }}"

When importing the test file test_create_scheduler.yml with the variable test_proxysql_scheduler_check_idempotence, the assert will check for status is not changed, because the ternary-filter evaluated the variable test_proxysql_scheduler_check_idempotence as true.

(Source)

Do things based on regex searches

In Ansible you can chain filters using a pipe (|). This allows you to filter based on regex searches (which in hindsight is obvious that it works, but I never thought about that).

- name: "{{ role_name }} | {{ current_test }} | are we performing a delete"
  set_fact:
    test_delete: "{{ current_test | regex_search('^test_delete') | ternary(true, false) }}"

(Source)

Handle older Python versions easily

In the following task, the cassandra-driver is installed. If the used (obsolete!) Python version starts with 2.7, pip should install the cassandra-driver in version 3.26.*. If a recent Python version is used, pip will install the latest version of the cassandra-driver.

- name: Install cassandra-driver
  pip:
    name: "cassandra-driver{{ ansible_python_version.startswith('2.7') | ternary('==3.26.*', '') }}"

That’s definitely not the most elegant solution, but it works. (I’d probably have tried to install the correct cassandra-driver version according to the operating system and its Python version, wher eit should be installed)

(Source)

Comment line in template if var is defined

This task will add a line starting with ssl_ciphers, if the variable zabbix_web_ssl_cipher_suite is defined and not none. Otherwise it will add the same line but commented out.

{{ (zabbix_web_ssl_cipher_suite is defined and zabbix_web_ssl_cipher_suite is not none) | ternary('', '# ') }}ssl_ciphers {{ zabbix_web_ssl_cipher_suite | default('') }}

(Source)

I used another way to add commented out lines in a template (see). I used the comment-filter:

{{ "HostKeyAlgorithms " ~ ssh_host_key_algorithms|join(',') if ssh_host_key_algorithms else "HostKeyAlgorithms" | comment }}

Now that I see my own code, I could probably use the ternary-filter here, too!

{{ ssh_host_key_algorithms | ternary("HostKeyAlgorithms " ~ ssh_host_key_algorithms|join(','), "HostKeyAlgorithms" | comment) }}

But I think I actually like the if-else syntax more.

Do you have any other interesting uses of the ternary-filter?

TIL how to create Files and Commits via the Github-API and Github-CLI

rndmh3ro — Mon, 11 Dec 2023 12:55:00 +0000

Recently, I found myself needing to incorporate a CODEOWNERS-file to multiple repositories within our Github organization. The objective was to automatically reviewers for pull-requests. And codeowners are a perfect tool for this. This way every colleague in our company can join our Github organization and get write-access to the repositories (so they can create branches) and finally create a pull requests without needing to fork it to their personal accounts.

But I absolutely wanted to avoid creating the codeowners-file by either:

Cloning all repositories before adding, committing, and pushing the files (much less through a Pull Request)
Adding the files through the Github UI (meaning clicking on “Add file”, “Create new File”, inserting the contents, and committing).

Fortunately you can also do this via Github’s REST-API! But it was harder than I thought!

I did not want to add individual persons to the codeowners-file - if the person leaves the company or does not want to be a codeowner anymore, I’d have to change the codeowners-file in all repositories again. Teams, in this context, are hugely beneficial. Github allows the creation of teams in your organization and you can people to them. You can then use these teams in the codeowners-file. This way, deleting a person from a team removes them as a codeowner across all repositories where the team is used.

After creating the teams and adding people to them (I did this by hand), I needed to give the teams access to the repositories. This I did not by hand but via the API.

The command to add a team to a repository looks like this:

> gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" -f permission=push -X PUT /orgs/telekom-mms/teams/terraform-maintainers/repos/telekom-mms/examplerepo; done

This grants the team terraform-maintainers access to the examplerepo-repository and grants them write-access (with -f permission=push).

Now if I want to do this for all terraform-repositories in our organization, I use a simple for-loop on the command-line, thereby searching for all repositories with gh repo list and then grepping only the repositories with terraform in their name:

> for i in $(gh repo list telekom-mms --json name -q .[].name -L 100 | grep terraform); do gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" -f permission=push -X PUT /orgs/telekom-mms/teams/terraform-maintainers/repos/telekom-mms/$i; done

Once the teams are added, I can then add them to the codeowners-file. To do this via the API, I basically have to PUT content into the repository (see the docs for more information). Since this is git after all, I need to do this in a commit. And a commit needs:

a commit message - this is done by adding the form-field message with the message as a value.
a committer - this way trickier to achieve. I need to put a json-string with a name and an email-address as
the content of the file - this is the content of the file as a base64-encoded string:

The final call to the API looks like this:

> gh api --method PUT repos/telekom-mms/examplerepo/contents/CODEOWNERS -F message="add codeowners" -F committer:='{"name:"Sebastian Gumprich",email:"sebastian.gumprich@telekom.de"}' -F content="KiBAdGVsZWtvbS1tbXMvdGVycmFmb3JtLW1haW50YWluZXJzCg=="; done

Do this in a loop and you’re good to go.

But of course you will be making an error when updating the file (at least I did) and you will then have to update the file. If you want to update a file using a commit, you cannot use the above command as-is. You need to add the blob-SHA of the file you want to replace.

How do you get it? By querying the API again (there’s a way to do this without using the API but I did not do this since the API-way was easier):

> gh api repos/telekom-mms/examplerepo/contents/CODEOWNERS -q .sha
d79b5f4ecfd52ef6ecea0a71744f6ce94a2522da

Now add this SHA to the API-call and then you can update the file:

> gh api --method PUT repos/telekom-mms/examplerepo/contents/CODEOWNERS -F message="update codeowners" -F committer:='{"name:"Sebastian Gumprich",email:"sebastian.gumprich@telekom.de"}' -F content="KiBAdGVsZWtvbS1tbXMvdGVycmFmb3JtLW1haW50YWluZXJzCg==" -F sha="d79b5f4ecfd52ef6ecea0a71744f6ce94a2522da"

With the correct owners in place, now the last step was to set the branch protection rule that defines that codeowners need to do reviews. This took me the longest time since from the documentation alone I couldn’t create a working request.

So I tried to use the existing branch protection of a repository (easily gettable by running gh api repos/telekom-mms/examplerepo/branches/main/protection), but getting the following code inside a working curl-request and then looping over said curl-request proved to be impossible:

{
  "url": "https://api.github.com/repos/telekom-mms/examplerepo/branches/main/protection",
  "required_pull_request_reviews": {
    "url": "https://api.github.com/repos/telekom-mms/examplerepo/branches/main/protection/required_pull_request_reviews",
    "dismiss_stale_reviews": false,
    "require_code_owner_reviews": true,
    "require_last_push_approval": true,
    "required_approving_review_count": 1,
    "bypass_pull_request_allowances": {
      "users": [],
      "teams": [],
      "apps": [
        {
          "id": 2740,
          "slug": "renovate",
          "node_id": "MDM6QXBwMjc0MA==",
          "owner": {
            "login": "renovatebot",
            "id": 38656520,
            "node_id": "MDEyOk9yZ2FuaXphdGlvbjM4NjU2NTIw",
            "avatar_url": "https://avatars.githubusercontent.com/u/38656520?v=4",
            "gravatar_id": "",
            "url": "https://api.github.com/users/renovatebot",
            "html_url": "https://github.com/renovatebot",
            "followers_url": "https://api.github.com/users/renovatebot/followers",
            "following_url": "https://api.github.com/users/renovatebot/following{/other_user}",
            "gists_url": "https://api.github.com/users/renovatebot/gists{/gist_id}",
            "starred_url": "https://api.github.com/users/renovatebot/starred{/owner}{/repo}",
            "subscriptions_url": "https://api.github.com/users/renovatebot/subscriptions",
            "organizations_url": "https://api.github.com/users/renovatebot/orgs",
            "repos_url": "https://api.github.com/users/renovatebot/repos",
            "events_url": "https://api.github.com/users/renovatebot/events{/privacy}",
            "received_events_url": "https://api.github.com/users/renovatebot/received_events",
            "type": "Organization",
            "site_admin": false
          },
          "name": "Renovate",
          "description": "[omitted for brevity]",
          "external_url": "https://www.mend.io/free-developer-tools/renovate/",
          "html_url": "https://github.com/apps/renovate",
          "created_at": "2017-06-02T07:04:12Z",
          "updated_at": "2023-11-06T09:25:36Z",
          "permissions": {
            "administration": "read",
            "checks": "write",
            "contents": "write",
            "emails": "read",
            "issues": "write",
            "members": "read",
            "metadata": "read",
            "packages": "read",
            "pull_requests": "write",
            "statuses": "write",
            "vulnerability_alerts": "read",
            "workflows": "write"
          },
          "events": [
            "issues",
            "pull_request",
            "push",
            "repository"
          ]
        }
      ]
    }
  },
  "required_signatures": {
    "url": "https://api.github.com/repos/telekom-mms/examplerepo/branches/main/protection/required_signatures",
    "enabled": false
  },
  "enforce_admins": {
    "url": "https://api.github.com/repos/telekom-mms/examplerepo/branches/main/protection/enforce_admins",
    "enabled": false
  },
  "required_linear_history": {
    "enabled": false
  },
  "allow_force_pushes": {
    "enabled": false
  },
  "allow_deletions": {
    "enabled": false
  },
  "block_creations": {
    "enabled": false
  },
  "required_conversation_resolution": {
    "enabled": false
  },
  "lock_branch": {
    "enabled": false
  },
  "allow_fork_syncing": {
    "enabled": false
  }
}

But I started using the above response and tried to cut it down to a minimal request that works. This took quite some time, but here’s the final result:

> curl -X PUT -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/telekom-mms/examplerepo/branches/main/protection -d '{
  "required_pull_request_reviews": {
    "dismiss_stale_reviews": false,
    "require_code_owner_reviews": true,
    "require_last_push_approval": true,
    "required_approving_review_count": 1,
    "bypass_pull_request_allowances": {
      "apps": ["branch-protection-as-code"]
    }
  },
  "enforce_admins": false,
  "restrictions": null,
  "required_status_checks": null
}'; done

Looping over this short piece of code worked. I now am able to manage our repositories at scale (now someone tell me the existing solution to do this).

How I teach Ansible to my colleagues: A hands-on training session.

rndmh3ro — Mon, 02 Oct 2023 20:00:00 +0000

As someone with years of experience using and teaching Ansible I want to share with you how I teach it to my colleagues in the most practical way possible.

However, before I get into the actual content of the training, it should be emphasized how important a functioning and identically set up working environment is for a successful learning journey.

A prerequisite for participation in the training is the possession of a work device with SSH client. The participants then use the SSH client to connect to virtual machines provided specifically for them. If these requirements are not met, problems can arise during the training that distract from the actual content I want to convey. Therefore, it is essential to prepare the training environment in advance to guarantee that everything runs as smoothly as possible. Sing Ansible, this is done almost fully automatically.

After laying down the prerequisites, let’s move on to the contents of the training. It is designed to last one day and covers the following topics: • What is Ansible and how does it work? • How to install Ansible? • How does Ansible work fundamentally and what components does it consist of? • What are ad hoc commands and how do you use them? • What are playbooks, how do you write them and how do you use them? • What are roles, how do you write them and how do you use them? • Where do you get Ansible roles, i.e. the ecosystem? • Best practices and practical insights for using Ansible

The training is a balanced mix of theory and practical exercises where the practical parts outweigh the theoretical.

In the first section, there is a short, theoretical overview of the purpose and functionality of Ansible as well, as a comparison with other tools and an insight into the ecosystem around the automation tool. This theoretical part takes a maximum of one hour, including a welcome and introduction of the participants.

To craft a personalized learning journey, I encourage learners to share their experiences, hopes, and expectations from Ansible during this introductory phase. This interactive exchange helps shape the session around the unique needs of each learner.

After setting the stage with a solid theoretical understanding, we leap into the world of practice.

I begin by demonstrating Ansible installation on various operating systems using a step-by-step approach. After that, I demonstrate the installation in my working environment. Finally, the participants perform the installation on their own in their working environment.

This method of ‘show-and-practice,’ which involves learners performing each step after its explanation and demonstration, is a constant presence throughout the session. It encourages immediate application of concepts, internalizes knowledge, and promptly addresses arising questions or issues.

I can quickly gather feedback and see whether all participants have understood everything.

When questions arise, even I, as an experienced user, occasionally reach the limits of my knowledge. However, if this happens, I can give the participants valuable methods on how to get to the solution of their questions, be it links to the right place in the documentation or the right search terms. It is important to show that lack of knowledge or the mistakes I make myself while using Ansible is not a bad thing and to live an open error culture.

I like to answer questions by showing and explaining code examples from practice. This way, the participants can immediately see that what they learn really has a practical use.

At the end of the training, there is still a summary of the best practices as well as the opportunity to clarify open questions or repeat things. After the training, I send the presentation to the participants and leave the working environment for a few more days so that the participants can once again independently apply and deepen the content they have learned and save their written code.

TIL how to create Azure Prometheus datasources with Ansible

rndmh3ro — Mon, 03 Jul 2023 08:00:00 +0000

Since I spent some time today on this, I’d rather write it down. Creating a Prometheus datasource that uses Azure Authentication was not straight forward.

Here’s the end result:

---
- name: Create a datasource in Grafana
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Create prometheus datasource
      community.grafana.grafana_datasource:
        name: prometheus_test
        ds_type: prometheus
        ds_url: https://example.westeurope.prometheus.monitor.azure.com
        url: "https://example.com"
        url_username: foo
        url_password: bar
        enforce_secure_data: true
        additional_json_data:
          azureCredentials:
            authType: clientsecret
            azureCloud: AzureCloud
            clientId: "{{ lookup('cloud.terraform.tf_output', 'clientid', project_path=playbook_dir + '../terraform/') }}"
            tenantId: "{{ lookup('cloud.terraform.tf_output', 'tenant_id', project_path=playbook_dir + '../terraform/') }}"
        additional_secure_json_data:
          azureClientSecret: "{{ lookup('cloud.terraform.tf_output', 'password', project_path=playbook_dir + '../terraform/') }}"

(Bonus: I lookup the client and tenant ID from Terraform state.)

How did I get to this? By creating the datasource by hand and then querying it via the Grafana API:

> curl -s 'https://example.com/api/datasources/7' | jq .
{
  "id": 7,
  "uid": "3E8CgP2Vk",
  "orgId": 1,
  "name": "Prometheus",
  "type": "prometheus",
  "typeLogoUrl": "",
  "access": "proxy",
  "url": "https://example.com.westeurope.prometheus.monitor.azure.com",
  "user": "",
  "database": "",
  "basicAuth": false,
  "withCredentials": false,
  "isDefault": false,
  "jsonData": {
    "azureCredentials": {
      "authType": "clientsecret",
      "azureCloud": "AzureCloud",
      "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
    },
    "httpMethod": "POST"
  },
  "secureJsonFields": {
    "azureClientSecret": true,
    "basicAuthPassword": true
  },
  "version": 10,
  "readOnly": false
}

There you get the jsonData and secureJsonFields. These are the special, required fields that you have to pass to Ansible to get exactly what you want.