DEV Community: Abdul Khaliq

Effortless NFS file transfer in CI/CD pipelines without Privileged access

Abdul Khaliq — Mon, 22 May 2023 15:23:59 +0000

In the realm of continuous integration and deployment (CI/CD) pipelines, the seamless transfer of artifacts to any remote servers holds immense significance.
Traditionally, in the case of NFS, achieving this demands mounting of the nfs share inside the runner, thereby restricting it to a specific node and granting privileged access.
However, NCP (NFS Copy), can be utilized to
allows direct artifact copying to remote NFS servers without the need for mounting, eliminating the constraints of privileged access.

In this article, we delve into the utilization of NCP in GitLab pipelines.

Understanding the Pipeline

Let's examine a sample pipeline to better understand how NCP can be leveraged for artifact transfer:

stages:
  - build
  - publish

build:
  stage: build
  script:
    - echo "build test artifact" > output.txt
  artifacts:
    paths:
      - output.txt
    expire_in: 2 hours

publish:
  stage: publish
  needs: ["build"]
  image: docker.io/khaliq/ncp:latest
  script:
    - ncp to --host 192.168.0.80 --nfspath data --input output.txt

The pipeline consists of two stages: build and publish. In the build stage, a test artifact is created by echoing the text "build test artifact" into a file called output.txt.

This stage is responsible for building the artifact that will later be transferred to the remote NFS server.

publish:
  stage: publish
  needs: ["build"]
  image: docker.io/khaliq/ncp:ncp
  script:
    - ncp to --host 192.168.0.80 --nfspath data --input output.txt

The publish stage is where the artifact is transferred using NCP. Here's a breakdown of the command used:

ncp to: Specifies the direction of the transfer, indicating that the artifact will be sent to the remote server.
--host: Specifies the IP address or hostname of the remote NFS server.
--nfspath: Defines the target path on the remote server where the artifact will be stored.
--input: Specifies the input artifact to be transferred, in this case, output.txt.

With this simple command, it handles the transfer of the artifact to the specified remote NFS server, without the need to mount nfs shares.

Conclusion

NCP simplifies remote NFS server integration in CI/CD pipelines. By eliminating nfs share mounting and the need for privileged access, it streamlines artifact transfer, granting the freedom to run jobs anywhere.

Checkout the GitHub repository for more information.

Installing gitlab on kubernetes with kustomize

Abdul Khaliq — Fri, 16 Jul 2021 08:14:04 +0000

Introduction

In this article, I will describe all the steps required to setup GitLab CI/CD in kuberntes using kustomize.
We will go through how to run GitLab on Kubernetes when you have related resources postgres, redis, minio, tls certificates etc already available in your setup.

This is a very common scenario in companies and also for self-hosting that you already have these services in your environment and prefer to use the same for gitlab.

The all in one production installation may be easily performed with Helm. You can refer to official documentation from gitlab if that is your requirement.

Requirements

You will need the following in order to run gitlab.

Database : Postgres database is required for gitlab.
Cache : Redis is used for caching.
Storage : Minio is used as object storage for container registry, gitlab backups, terraform storage backend, gitlab artifacts etc.
Ingress Controller : Nginx ingress is part of installation.
Persistent Volume : Gitaly will store repository data data on disk, for that your kubernetes cluster must have a way of provisioning storage. You can install local path provisioner in your cluster for dynamically provisioning volumes.

Repositories
1. Gitlab Manifests
2. SubVars App

Info: You can swap minio with any other object storage i.e S3 by changing connection info secret

Lets get Started

When installing gitlab with helm it generates the configmaps after rendering the templates with parameters, we can manually change these values in configmaps but its a hassle and not convenient.

To make this process easy we will use a tool called subvars which will let us render values defined as go templates from command line. Install it by following the instructions on github page, we will use it later.

Download the release with manifests from github alternatively you can clone the repo, if you are cloning the repo remove the .git folder afterwards as it creates issues some times when rendering multiple version of the same file with subvars.

export RELEASE_VER=1.0
wget -q https://github.com/kha7iq/gitlab-k8s/archive/refs/tags/v${RELEASE_VER}.tar.gz
tar -xf v${RELEASE_VER}.tar.gz
cd gitlab-k8s-${RELEASE_VER}

Lets start by setting the url for gitlab in our kustomization file within ingress-nginx folder. You will find two blocks, one for web-ui and second for registry along with tls-secret-name for https.

patch: |-
    - op: replace
      path: /spec/rules/0/host
      value: your-gitlab-url.example.com
    - op: replace
      path: /spec/tls/0/hosts/0
      value: your-gitlab-url.example.com
    - op: replace
      path: /spec/tls/0/secretName
      value: example-com-wildcard-secret

We can create minio-conn-secret containing configuration for minio. It will be used for all the enabled S3 buckets except gitlab backups, we will create that separately. Input the information as per your setup and create the kubernetes secret.

minio.config

cat << EOF > minio.config
provider: AWS
region: us-east-1
aws_access_key_id: 4wsd6c468c0974006d
aws_secret_access_key: 5d5e6c468c0974006cdb41bc4ac2ba0d
aws_signature_version: 4
host: minio.example.com
endpoint: "https://minio.example.com"
path_style: true
EOF

Kubernetes secret

kubectl create secret generic minio-conn-secret \
--from-file=connection=minio.config --dry-run=client -o yaml >minio-connection-secret.yml

Next step is to create a secret with mino configuration for gitlab backup storage. Just replace minio endpoint, bucket name, access key & secret key.

cat << EOF > storage.config
[default]
access_key = be59435b326e8b0eaa
secret_key = 6e0a10bd2253910e1657a21fd1690088
bucket_location = us-east-1
host_base = https://minio.example.com
host_bucket = https://minio.example.com/gitlab-backups
use_https = True
default_mime_type = binary/octet-stream
enable_multipart = True
multipart_max_chunks = 10000
multipart_chunk_size_mb = 128
recursive = True
recv_chunk = 65536
send_chunk = 65536
server_side_encryption = False
signature_v2 = True
socket_timeout = 300
use_mime_magic = False
verbosity = WARNING
website_endpoint = https://minio.example.com
EOF

kubectl create secret generic storage-config --from-file=config=storage.config \
--dry-run=client -o yaml > secrets/storage-config.yml

All other secrets can be used as is from repository or you can change all of them following gitlab documentation

One of the most important secret is gitlab-rails-secret, in case of a disaster where you have to restore gitlab from a backup you must apply the same secret in your cluster as these keys will be used to decrypt the database etc from backup. Make sure you keep this consistent after first install.
We reached the last part, Its alot of work to change database and other parameters one by one in configmaps.
I have implemented some templating for this we can provide all the values via environment variables and render the manifests with subvars, it will output these to destination folder and replace all the parameters defined as go templates.

Variables are self explanatory, GITLAB_GITALY_STORAGE_SIZE variable is used to specify how much storage is needed for gitaly and GITLAB_STORAGE_CLASS is the name of storage class.

GITLAB_URL=gitlab.example.com \
GITLAB_REGISTRY_URL=registry.example.com \
GITLAB_PAGES_URL=pages.example.com \
GITLAB_POSTGRES_HOST=192.168.1.90 \
GITLAB_POSTGRES_PORT=5432 \
GITLAB_POSTGRES_USER=gitlab \
GITLAB_POSTGRES_DB_NAME=gitlabhq_production \
GITLAB_REDIS_HOST=192.168.1.91:6379 \
GITLAB_GITALY_STORAGE_SIZE=15Gi \
GITLAB_STORAGE_CLASS=local-path \
subvars dir --input gitlab-k8s-1.0 --out dirName

Change into dirName/gitlab-k8s-1.0 you can have a look to confirm if everything is in order before applying this in cluster.

The final step is to create the namespace gitlab and build with kustomize or kubectl. I prefer kustomize but you can also use kubectl with -k flag.

Create namespace

kubectl create namespace gitlab

Apply the final manifest

kustomize build gitlab-k8s-1.0/ | kubectl apply -f -
# or following if you have already changed into directory
kustomize build . | kubectl apply -f -

# With kubectl
kubectl apply -k  gitlab-k8s-1.0/
# or following if you have already changed into directory
kubectl apply -k  .

Head over to the endpoint you have configured for gitlab https://gitlab.example.com and login.

Info: Default passwords
- Gitlab 'root' user's password

   LAwGTzCebner4Kvd23UMGEOFoGAgEHYDszrsSPfAp6lCW15S4fbvrVrubWsua9PI

Postgres password configured as secret

    ZDVhZDgxNWY2NmMzODAwMTliYjdkYjQxNWEwY2UwZGMK

Performing code checks with GitHub actions & sending results to slack

Abdul Khaliq — Thu, 29 Apr 2021 19:15:55 +0000

What is a GitHub Action ?

A GitHub action is an element of your custom workflow that is triggered when a push or pull request is created and performs some action such as checking code, sending a message or deploying the application.

The most important concept of GitHub actions is their composability. You can assemble actions as building blocks and build a workflow that matches your needs.

In this article we would like to explore how we can put certain actions together and perform different steps based on outcome.

Lets create an action which performs linting checks on a new pull request.

on: pull_request

jobs:
  lint-code:
    runs-on: ubuntu-latest
    name: Perform Checks
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: golangci-lint
        uses: golangci/golangci-lint-action@v2
        with:
          version: v1.29

Sending alerts

Next we want to get a notification on slack if these checks passed or failed.

We will use PingMe Action to send these alerts.

on: pull_request

# Get values from github secrets of slack token and target  channels and set the variables.
# we can set these within the action block as well for pass/fail.
env:
  SLACK_TOKEN: ${{ secrets.SLACK_TOKEN }}
  SLACK_CHANNELS: ${{ secrets.SLACK_CHANNELS }}

jobs:
  lint-code:
    runs-on: ubunut-latest
    name: Perform Checks
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      # We want to perform checks and see if the code is properly linted.
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v2
        with:
          version: v1.29

      - name: Alert when checks fail
        uses: kha7iq/pingme-action@v1
        # This action will only run if checks failed.
        if: failure()
        env:
          SLACK_MSG_TITLE: '🟢 New Request: ${{ github.ref }}'
          SLACK_MESSAGE: 'Event is triggerd by ${{ github.event_name }} Checks  ❌  ${{ job.status }}'
        with:
          service: slack

      - name: Alert when checks pass
        uses: kha7iq/pingme-action@v1
        # This action will only run if checks are successfull.
        if: success()
        # Message and Title are string values, you can create custome message or title.
        env:
          SLACK_MSG_TITLE: '🟢 New Request: ${{ github.ref }}'
          SLACK_MESSAGE: 'Event is triggerd by ${{ github.event_name }} Checks ✅  ${{ job.status }}'
        with:
          service: slack

Passed
Failed

You can see this workflow in action here.

kha7iq / pingme

PingMe is a CLI which provides the ability to send messages or alerts to multiple messaging platforms & email.

PingMe CLI

Documentation • Supported Services • Install • Github Action • Configuration • Contributing •

About

PingMe is a personal project to satisfy my needs of having alerts, most major platforms have integration to send alerts but its not always useful, either you are stuck with one particular platform, or you have to do alot of integrations. I needed a small utility which i can just call from my backup scripts, cron jobs, CI/CD pipelines or from anywhere to send a message with particular information. And i can ship it everywhere with ease. Hence, the birth of PingMe.

Everything is configurable via environment variables, and you can simply export the logs or messages to a variable which will be sent as message, and most of all this serves as a swiss army knife sort of tool which supports multiple platforms.

Supported services

Discord
Email
Gotify
Line
Mastodon
Mattermost
Microsoft…

View on GitHub

Send alerts to multiple platforms from your scripts

Abdul Khaliq — Tue, 27 Apr 2021 10:32:37 +0000

In this article, we will illustrate how to configure and send messages to multiple platforms from your scripts.

A notification can be setup based on success or failure of a task or if you want to have an alert when a certain condition is meet.

This article uses a small binary PingMe to send this notification, you can find more information about it on github.

Lets dig right into it

In this first example we would like to get a notification when any one logs into our server via ssh.
Multiple options are available for messaging platform in this example we will use Pushover service.

Download & install

First step is to download install the binary itself on the system which we will use for sending notifications.
We can download the binary from github and extract in the current folder.

Its executable but for good measure we will make it executable and then move the binary in system path, in this case its /usr/local/bin.

App is also available as deb and rpm, for windows users a compatible binary is can be found under release page.

wget -q https://github.com/kha7iq/pingme/releases/download/v0.1.5/pingme_Linux_x86_64.tar.gz && tar -xf pingme_Linux_x86_64.tar.gz
chmod +x pingme
sudo mv pingme /usr/local/bin/pingme

Lets create our script and configure notifications

Send Alert when any user login via ssh

/etc/profile.d/ is read at every login (for bash shell users). The if statement will only return true if the user has logged in via ssh, which in turn will cause the indented code block to be run.

Create a file with desired name i.e /etc/profile.d/login.sh and paste the following.

pushover_token="<your token here>"
pushover_user="<your user here>"
alert_title="Alert SSH Login: $(date)"
if [ -n "$SSH_CLIENT" ]; then
    PUSHOVER_MESSAGE="ssh login to ${USER}@$(hostname -f) from $(echo $SSH_CLIENT|awk '{print $1}')"
    pingme pushover --msg "$PUSHOVER_MESSAGE" --token $pushover_token --user $pushover_user --title "$alert_title"
fi

Pushover Alert

Send alert on failure or success of a cronjob

We have a cronjob which runs and take backup of our files, we would like to know if this backup has been created successfully
or failed. In this example we are going to send the alert to telegram channel.

Cron Job

* * * * * /home/user/scripts/backup.sh

Backup Script

#!/bin/bash
# telegram bot token
token="<token here>"
# Telegram channel
channel="<channel here>"
DATE=$(date +%d-%m-%Y-%T)
BACKUP_FILE_NAME="postgres_db-$DATE.tar.gz"
BACKUP_DIR="/home/user/backup"

tar -zcpf $BACKUP_DIR/$BACKUP_FILE_NAME $HOME/dbfiles

# check the exit code and send alert based on that.
if [ $? -eq 0 ]
then
    title="Success: Backup created successfully"
    /usr/local/bin/pingme telegram --msg "Successfully backed up $BACKUP_FILE_NAME" -t $token -c $channel --title "$title"
else
    title="Failed: Error creating backup"
    /usr/local/bin/pingme telegram --msg "Unsuccessful backup of $BACKUP_FILE_NAME" -t $token -c $channel --title "$title"
fi

Telegram Alert