DEV Community: David Bros

5 tips to make your Gitlab CI/CD cleaner

David Bros — Mon, 27 Feb 2023 11:16:36 +0000

Gitlab can be a bit of a challenge if you have never written any CI/CD pipelines, forget about your software development mindset and quickly adapt to the pipeline mindset with these 5 easy to remember tips.

Templates!
Abstract Logins
Keep jobs simple
Keep jobs neatly linked
Keep important steps manual

Templates!

Pipelines can get messy, their top-down execution means you need to carefully analyze which rules apply to each job before executing them. You can simplify most of these rules with templates.

Templates can be defined in separate repos, and then used in your project's gitlab-ci.yml for extra commodity and abstraction. You can enable this behavior with include and extends

ci-cd-templates repo
templates.yml

.echo-statement
- before_script:
  - echo "this is an inherited step!"

my-project repo
gitlab-cy.yml

include:
  - project: 'ci-cd-templates'
    file: 'templates.yml'
    rev: (tag or branch)

example-job:
  extends:
    - .echo-statement
  script:
    - echo "this is a normal step!"

Important caveat

Commands in your job will be defined by the template, if you define them in example-job they will overwrite the template's keys and values. This is my templates normally use a before/after script.

Abstract Logins

Let's use the freshly acquired knowledge from the last tip to simplify our pipelines. When doing pipelines you will inadvertably reach a point where you will need to login into something, cloud, api, gitlab api, anything really.

Keep your pipelines cleaner by abstracting the logins to your templates. Since logins almost always generate files from which to yield the access tokens, combine this with artifacts and abstract your logins.

ci-cd-templates repo
templates.yml

.aws-login
variables:
  role: MY-ROLE
  role_session_name: dev
  account_id: 0000000
before_script:
  - aws sts assume-role --role-arn "arn:aws:iam::${ACCOUNT_ID}:role/${ROLE}" --role-session-name "${ROLE_SESSION_NAME}" > creds.tmp.json
  - echo AWS_ACCESS_KEY_ID=$(cat creds.tmp.json | jq -r ".Credentials.AccessKeyId") > build.env
  - echo AWS_SECRET_ACCESS_KEY=$(cat creds.tmp.json | jq -r ".Credentials.SecretAccessKey") >> build.env
  - echo AWS_SESSION_TOKEN=$(cat creds.tmp.json | jq -r ".Credentials.SessionToken") >> build.env
  - rm creds.tmp.json
artifacts:
  reports:
    dotenv: build.env

Attach this login templated job to your deployment jobs.

deploy-dev:
  extends: .aws-login
  script:
    - docker build -t [...]
    - docker push [...]

Keep jobs simple

When developing jobs, keep them as purpose fit as possible, this will ensure readability and maintainability.

Declare the same stages every time and give them individual stages. Those who have read "The Pragmatic Programmer" will kill me for saying this, but remember: forget about your programmer mindset.

A template we use in almost every project:

gitlab-ci.yml

stages:
- test
- build
- deploy

make-test:  # executes unittests / integration tests
  stage: test

init-dev-dependencies: # downloads dependencies
  stage: build

build-dev: # builds a docker image / gz / tar / etc
  stage: build

deploy-dev: # deploys / uploads a docker image / gz / tar / etc
  stage: deploy

init-tst-dependencies: # same steps, different enviornment
  stage: build

build-tst:
  stage: build

deploy-tst:
  stage: deploy

[ other environments ]

Keep jobs neatly linked

Neatly tie jobs in order to form dependent pipelines with needs and dependencies.

Needs: Needs a job to run before itself, this will create independent pipelines for each environment, if you define it correctly.
Dependencies: Depends on the files from the dependent jobs.

gitlab-ci.yml

make-test:
  stage: test

init-dev-dependencies:  
  stage: build
  needs: 
    - make-test

build-dev:
  stage: build
  needs:
    - init-dev-dependencies
  dependencies: 
    - init-dev-dependecies

deploy-dev:
  stage: deploy
  needs:
    - build-dev
  dependencies:
    - build-dev

init-tst-dependencies:
  stage: build
  needs:
    - make-test

build-tst:
  stage: build
  needs:
    - init-tst-dependencies
  dependencies:
    - init-tst-dependencies

deploy-tst:
  stage: deploy
  needs: 
    - build-tst
  dependencies:
    - build-tst

For a clearer view of what needs actually does:

With needs:

Without needs:

So, in essence, it parallelizes.

Keep important steps manual

Lastly, keep the most important or dangerous steps manual, by adding when.

When will make sure a user presses the button in the gitlab UI in order for the step to run. Of course jobs that are linked to a manual job via needs will not run until the button is pressed. It is also highly recommended to tie the initial production job to the acceptance / test deployment, for extra security against unwanted deployments.

gitlab-ci.yml


init-prd-dependencies:  # no need for unit tests when deploying to prod, since it should already be tested in the dev/tst/acc pipelines
  stage: build
  when: manual

build-prd:
  stage: build
  needs:
    - init-prd-dependencies
    - deploy-acc
  dependencies: 
    - init-prd-dependecies

deploy-prd:
  stage: deploy
  needs:
    - build-prd
  dependencies:
    - build-prd

These are some tips from things we have learned in the past months, these GitLab features have helped our team create better, safer, readable and robust pipelines.

Mastering Step Functions States and Paths

David Bros — Sun, 04 Dec 2022 11:01:22 +0000

AWS has gained exponential traction over the past years, and for all the good reasons.

While as a beginner AWS can be a bit of a headache (you know, due to the fact that there is over 200+ services to choose from) the entry level for the grand majority of them is relatively low, especially you have some software engineering experience.

So is the case for Step Functions, easy to understand, relatively hard to master. In this article I want to guide you through some techniques and introduce you to some concepts that will help you understand and efficiently manage the state.

Intro

Wait, what are step functions?

Step functions are infrastructure as code meant to represent the series of steps your logical process. This process can be anything, moving data from S3 buckets, executing Lambdas, sending Alarms. Any of the AWS Services can be used in custom made flows that you will define, and later have available as IAC.

In this article, rather than explain what you can do with them, I will detail what the state is and how to manage it.

Set Up (Only if you want to follow along practically)

AWS offers free tier for 12 months (with certain limits), if you have never developed Step Functions in AWS, I recommend you build the step function as you follow along the article.

Make sure the step function has rights of execution by adding the AWSLambdaRole policy to the Step Function's role.

The State

The state is a JSON object that defines the input of the logical step you are executing as well as later becoming the output of it. This JSON structure is completely customizable, however, you need to take care of it's structure and content, otherwise it will become unreadable and unmaintainable.

Types of state:

Execution State (Execution Context)
Local State (Local Context)

Execution state

The execution state is the input the step function was executed with.

It can be defined when executing the step function manually or by schedulers/orchestrators which do it for you. You can modify it, but it is highly recommended you don't, this way you will be able to know how the step function was executed..

The execution state will be accessible during the whole flow of the function, leverage it!

Accessing the execution state

By using double $ as a prefix you can access the Execution State and use it's keys as if they were variables. You will also need to use .$ as a suffix in the key where you want to use it.

For example, let's run the hello-world Lambda using the following:

Execution State (Execution Input)



{
  "key1": "This is an execution argument!",
  "key2": "No, I can't fix your printer!",
  "key3": "HTML is not a programming language!"
}

Lambda Payload:



{
  "key1.$": "$$.Execution.Input.key1",
  "key2.$": "$$.Execution.Input.key2",
  "key3.$": "$$.Execution.Input.key3",
}

Remember:

.$ as a suffix in the keys tells step functions that the value will be a call to a State.
$$. as a prefix tells the step functions to access the Execution state.

Lambda step should look like this now:

Run the function:

If you look at the results, you will see the values have been successfully replaced:

Local State

The local state is the one you get to play with the most, since it gets changed / manipulated in every step.

In the first step of the step function, the Execution State will be copied and used as an input.

The flow goes as follows:

local state is used as input for a step (task) > local state is now the output of that step (task)

However, sometimes you want to be able to keep the output of a step for another one which is further down the line.

Whatever the case is, in 90% of situations you will need to manipulate it in order to fit your flow's needs.

Left unchecked, your state will become messy and unreadable (truncated, some of these objects are very long):

In this picture, the lambda invoke is the first step, thus the input is a copy of the Execution State. After that, the local state will become the Output. In the next step, the Input would be the Output shown.

Manipulating the local state

You can manipulate the state with the given paths:

InputPath
ResultSelector
OutputPath
ResultPath

It is key you understand what all these paths mean in order to master manipulating your state and leveraging it.

All the paths can be enabled at every step of your flow:

Input:

Output:

How are they executed:

They are optional, so you can activate or deactivate them at any point, creating any combination with them.
The flow goes as follows, this is the case for every step in your flow:

InputPath > Step Execution (Lambda) > ResultSelector > ResultPath> OutputPath

InputPath

The InputPath takes care of filtering the local state that acts as input for a certain step, you can select a certain key to use as input for the step, instead the whole local state that was passed from the previous step.

Add the following to the InputPath: ```

$.key1


Example:

![Filtering Input](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0lkaz5dd4jvtav41vjdu.png)

It looks like nothing happened, activate the advanced view:

![Remmaped payload](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x3yl1iw6o49ozq2pv9wd.png)

The payload was successfully changed. (The Lambda will fail, since the input is not supposed to be a string).

#### ResultSelector <a name="result-selector"></a>

The ResultSelector takes care of filtering and transforming the step's output, with this you can shape the output to whatever form you want.

Add the following to ResultSelector:

{
"im_changing_the_field.$": "$",
"im_a_copy.$": "$",
"im_another_copy.$": "$.Payload"
}


Run the function, look at the results (truncated):

![Results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xx4yf1wyyv86hr9z4mqn.png)

#### ResultPath <a name="result-path"></a>

The ResultPath (different than ResultSelector) takes care of adding or substracting structures from your state. This step is limited to two options:

- Add the output from OutputPath to your original unfiltered input with a new key.
- Discard the output from OutputPath and return your original unfiltered input. 

Copy the following to your ResultPath (disable ResultSelector): ```

 $.lambda_result 

``` and select the first option. 

Look at the results:

![Results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pn7a0rpai7qgl9mjq3la.png)

Do the same with the second option and look at the results:

![Results-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/puxa6c0vhykoakz98bn4.png)

#### OutputPath <a name="output-path"></a>

The OutputPath does the same as the InputPath, but for the output. Take in account this output <u>can be the raw output from the step or, a transformed output that comes from ResultSelector/ResultPath</u>.

Add the following to OutputPath (disable ResultsPath): ```

 $.Payload

Run the function, look at the results (enable advanced view):

Notice how the Result was filtered by the OutputPath.

Now try to combine all of them at the same time ;)

End Note

Now you know how the state works and how to manipulate it, it falls on your shoulders to keep it clean and readable.

Reflect on the following:

What does my previous state output, is all the output needed for the next step?
What does my next step need as an input, is all the local state needed for it?
Do I need my local state to output at the end of the function, can you discard it?
Will my step function have an execution input? How can you leverage that in order to keep the local state clean and simple?

These questions will help you design clean and readable step functions.

Thank you for reading and see you in the next one!

ELK: Logstash in 5 minutes

David Bros — Tue, 22 Mar 2022 23:19:18 +0000

This is a continuation to the previous article: Elasticsearch in 5 minutes.

Logstash is the second essential tool for Data Engineering. Like Elasticsearch it is developed and maintained by the Elastic team and it also has a free version available (which will cover all your needs).

In this article we will learn what Logstash is and how it works, we will also set up a Logstash processor and play with some of the plugins it has to offer. All in under 5 minutes!

For this article I have set up a Github Repository where you can find all the files needed to complete some of the steps as well as the images within this article.

What is Logstash
Logstash is a data processor, it acts as the pipeline between your databases or nodes and other resources, which could be other databases, nodes or applications.

However the true power of Logstash comes in its data manipulation functions: Transforming and Filtering operations.

Where Logstash Shines

Data Processing
Data Streaming Enviornments
Data Analytics Environments

Why does it shine in these enviroments

High compatibility with input and output resources: Logstash can read from almost anything, including Cloud applications, message Brokers and websockets. Here is the list for input and output plugins.
Fast and flexible transformation functions: Transform data structures with simple JSON defined operations.
Filtering: Huge range of filtering functions are your disposal, you can use this in different pipelines to create multiple sources from one raw source.

What version are we setting up
We will be setting up a free Logstash processor version 8.x

Step 1, Docker
For ease of use we will be using a Centos7 Docker container.

Pull the image:
docker pull centos:7
You can find the image here

Run the container with SYS Admin permissions and a mounted volume:
docker run -id --cap-add=SYS_ADMIN --name=logstash-centos7 -v /sys/fs/cgroup:/sys/fs/cgroup:ro centos:7 /sbin/init

We run the container with SYS ADMIN permissions because we need systemctl to work inside the container

Connect to the container:
docker exec -it logstash-centos7 /bin/bash

Step 2, Machine Setup
Update the machine:
yum update -y && yum upgrade -y

Install a text editor (any):
yum install vim -y

Step 3, Install Logstash

Install the GPG key for elastic:
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add this logstash repo under /etc/yum.repo.d/logstash.repo:

[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Remember to save!

Download and install the repo
yum install logstash

Start logstash
systemctl start logstash

Verify logstash is running
systemctl status logstash

Now stop logstash, we'll restart it later down the line
systemctl stop logstash

Step 4, set up an elastic node

We will be using Elasticsearch as input for our pipelines, so you will need to set up another Docker image with an elastic node, you can find a guide here if you don't have one already running.

Annotate the IPs of your containers

By default Docker sets up a network called bridge which containers can use to communicate between themselves, you can see the IPs by first seeing your container IDs, then inspecting the container:

docker ps

docker inspect {docker container id}
Docker inspect will yield a very large JSON, search for the field IPAddress

Annotate the IPs for both containers, we'll use them in the next step.

Create a test_pipeline index and insert sample data, you need to do this inside your Elasticsearch container. You can copy this sample data and paste it under /tmp/sample_data.json

vim /tmp/sample_data.json

Check the index has been created correctly

curl -X GET --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200/_cat/indices

Index this sample data we previously saved

curl -X POST --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic -H 'Content-Type: application/nx-ndjson' https://localhost:9200/_cat/indices --data-binary @/tmp/sample_data.json

Create another index called test_pipeline_output

Due to time constraints (hint on the title) we are going to take some shortcuts: Disabling SSL and Setting our replicas to 0.

To do disable ssl you need only change xpac.security.enabled to false under the file /etc/elasticsearch/elasticsearch.yml

Restart elasticsearch
systemctl restart elasticsearch

Now set your replicas to 0, do this step for both test_pipeline and test_pipeline_output indices:
curl -X PUT -u elastic http://localhost:9200/**test_pipeline**/_settings -H 'Content-Type: application/json' --data '{"index": {"number_of_replicas": 0}}'

Step 5, set up a logstash pipeline

Copy this configuration under
/etc/logstash/conf.d/pipeline.conf

The purpose of this pipeline is to get the messages from index test_pipeline to another index called test_pipeline_output

The only thing left now is to restart logstash in our logstash docker container.

systemctl restart logstash

Let it run for a couple of minutes, after that check indices in your Elasticsearch node:

curl -X GET --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic http://localhost:9200/_cat/indices/

In the image you can see that the output index has 100 items, while the original has 10, this is because logstash will continuously run this pipeline every few seconds.

In any case, with this article we have successfully set up a Logstash processor and moved data around with a very basic setup.

Follow me for part 3, where we'll set up a Kibana UI in 5 minutes!

ELK: Elasticsearch in 5 minutes

David Bros — Tue, 15 Mar 2022 20:59:23 +0000

If you claim to be a Data Engineer or a DevOps engineer then you surely have heard of Elasticsearch, this technology has been gaining traction for all the right reasons.

Although you can use Elasticsearch only for storage, it is highly recommended to use it alongside it's brothers: Logstash and Kibana, we will get to those in other posts.

What is Elasticsearch
Elasticsearch is a NoSQL document database (JSON), its environment revolves around 3 main concepts:

Nodes: An Elasticsearch node is a server which stores data, it does this through shards.
Shards: Shards are a storage abstraction that provides easier access to your data as well as support for replication.
Indices: Indices are the second storage abstraction, they are what you are going to interact with when fetching or aggregating data from the Elasticsearch API.

![Elastic Node](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vnizubkxockvhpi2rqsv.png)

Where Elasticsearch shines:

Data Streaming Environments: Together with Logstash
Data Analytics Enviornments: Together with an Analytics Frontend such as Kibana
Data Lake: Use the Elasticsearch REST API, compatible with any data analytics processing application.

Why does it shine in these environments:

Fast and efficient I/O operations: Provided by sharding and indexing.
Aggregation support: Elasticsearch has an incredibly flexible REST API that you can use from any client to make custom aggregations and data structures.
Kibana: Kibana is a front end application that fully integrates with Elasticsearch, it uses the Elasticsearch REST API to support all sorts of graph visualizations.

What version are we setting up
Elasticsearch has a payed and free version, you'll learn how to set up the free version.

Step 1, Docker:
For ease of usage, let's use Docker. As mentioned in the beginning, we'll use docker and CentOS 7.

Pull the image:
docker pull centos:7
You can find the image here

Run the container with SYS Admin permissions and a mounted volume:
docker run -id --cap-add=SYS_ADMIN --name=elasticsearch-centos7 -v /sys/fs/cgroup:/sys/fs/cgroup:ro centos:7 /sbin/init

We run the container with SYS ADMIN permissions because we need systemctl to work inside the container

Connect to the container:
docker exec -it elasticsearch-centos7 /bin/bash

Step 2, Machine Setup
Update the machine:
yum update -y && yum upgrade -y

Install a text editor (any):
yum install vim -y

Step 3, Install Elasticsearch

Install the elasticsearch repo under /etc/yum.repos.d/elasticsearch.repo:
vim /etc/yum.repos.d/elasticsearch.repo

Paste the following text:

[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

Remember to save!

You can find the latest version of this configuration here

Install Elasticsearch:
yum install --enablerepo=elasticsearch elasticsearch -y

There are other options to download the repository, but this is the easiest.

Reload the daemon
systemctl daemon-reload

Enable and Start elasticsearch:

systemctl enable elasticsearch && systemctl start elasticsearch

Step 4, Verify

Check the service is running
systemctl status elasticsearch

Reset your password for user elastic
/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Confirm that elasticsearch works by querying the Elasticsearch API (https enabled by default):
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

You are now the proud owner of a free ElasticSearch Node!

Follow me for part 2, where we'll set up Logstash in 5 minutes!

DEV Community: David Bros

5 tips to make your Gitlab CI/CD cleaner

Templates!

Abstract Logins

Keep jobs simple

Keep jobs neatly linked

Keep important steps manual

Mastering Step Functions States and Paths

Table of contents:

Intro

Set Up (Only if you want to follow along practically)

The State

Execution state

Local State

Manipulating the local state

InputPath

End Note

ELK: Logstash in 5 minutes

ELK: Elasticsearch in 5 minutes