DEV Community: hatem ben tayeb

Looking for recommendations for provisioning a production AKS cluster

hatem ben tayeb — Mon, 18 Jan 2021 18:03:18 +0000

Scale your containers with ansible on a non kubernetes/swarm environment

hatem ben tayeb — Sat, 16 Jan 2021 19:23:35 +0000

Scaling containers on Kubernetes with Replicatsets or scale services with docker swarm is a very easy task! but what about scaling containers in a non managed environment, when you have high traffic, absolutely you need to scale and load balance the traffic among multiple containers. In this article, we will use Ansible and nginx to scale containers.

Original post: here

Bash: writing a simple pod checker

hatem ben tayeb — Fri, 15 Jan 2021 18:52:52 +0000

I was working with Kubernetes and I just want to check my pods in the current workspace, in a funny way 😅, I will use bash for just one reason, it's the Linux native languages no need for other languages and libraries

Actually, I was inspired by a cool tool called popeye, you can find it here.

Make sure you have Kubectl installed and an existing cluster as well, here is my implementation

Define your favorite colors

blanc="\033[1;37m"
gray="\033[0;37m"
magento="\033[0;35m"
red="\033[1;31m"
green="\033[1;32m"
amarillo="\033[1;33m"
azul="\033[1;34m"
rescolor="\e[0m"

You can find more about colors in bash here.

Get list pods into an Array

listPods=$(kubectl get po | awk 'NR>1{print $1}')
#echo "$listPods"
readarray  arr <<<  $listPods

NR>1 will skip the first line and print $1 will print the first words (separated by a space) on all lines.

Looping over the array and check the status

ok=0
notok=0
echo -e "\nSit Down and Wait  \U1F602 :\n"
for i in ${arr[@]}
do 
echo -ne "$i ... " 
status=$(kubectl get po $i | grep $i | awk '{print $3}')
    if [[ ! $status =~ ^Running$|^Completed$  ]]  ; then
        echo -e "\e[1;31mOh Shit !"$rescolor""
        notify-send "Pods Health" "$i was  FUCKED" -t 10000 
        let notok=notok+1
    else
        echo -e "\e[1;32mOK!"$rescolor""
        #notify-send "Pod $i Is Good :)"
      let ok=ok+1
    fi
done

The ok and notok are used to count the number of the running/not running pods , the ${arr[@]} prints out the whole array, the notify-send will create a notification on your system is one of the pods are fked up** .

Print out the summary

echo -e "\nSTATS:\n"
echo "+---------------+---------------+"
printf  "|$green%-15s$rescolor|$red%-15s$rescolor|\n" "Healthy Pods" "Unhealthy Pods"
echo "+---------------+---------------+"
printf  "|%-15s|%-15s|\n" "$ok" "$notok"
echo "+---------------+---------------+"
echo -e "\n"

Run the script

Download the scripts with CURL :

curl https://raw.githubusercontent.com/hatembentayeb/podschecker/master/podschecker.sh --output podschecker.sh
chmod +x podschecker.sh

Demo

Repository: https://github.com/hatembentayeb/podschecker
Original blog: https://hatembentayeb.hashnode.dev/bash-writing-a-simple-pod-checker

The scripts don't do much but it was a result of a boring 3 hours on this pandemic 🥲

Heroku: Deploy a dockerized Flask ML app

hatem ben tayeb — Tue, 12 Jan 2021 19:49:43 +0000

In this post we will deploy a simple dockerized Machine learning application written with the Flask micro-framework to the Heroku (PaaS), using one of the leading CI/CD tools, which is Gitlab!, all used tools are free and no fees in achieving this tutorial.

Original article: https://hatembentayeb.hashnode.dev/heroku-deploy-a-dockerized-flask-ml-app

Naniiiiiiiiii !!

hatem ben tayeb — Tue, 12 Jan 2021 13:17:24 +0000

I was inspired by several readme profiles on Github and especially this awesome repository: https://github.com/coderjojo/creative-profile-readme
and finally, I created my own https://github.com/hatembentayeb

Share your readme profile as a comment 😋😋😋

Hiding my nodejs application code within a docker container

hatem ben tayeb — Mon, 11 Jan 2021 02:50:39 +0000

Building binaries on compiled languages like Rust and Go as Static or dynamic linking makes a huge step on code security, i mean no one can access your code, reading it !, but in many use cases they can by doing some complicated reverse engineering methods ... actually it's hard ! it become more harder when you built your app as a static linking nd bundle all the shit 💩 together. Nodejs applications can be read it easly ... all your source code is accessible, In this post we will hide our node app and "COMPILE" it ! seems owesome right ! let's go folks 😇 !

Original post : https://hatembentayeb.hashnode.dev/hiding-my-nodejs-application-code-within-a-docker-container

Mailcow: Setting up a full featured self hosted mail server

hatem ben tayeb — Sun, 10 Jan 2021 15:10:02 +0000

Setting up a full featured mail server is an actual PAIN for a system admin, you have to interconnect many software pieces and a lot of testing espacially security and spam if you plan to use it for your company ... In this Post i will take you in my journey of setting up a full featured mail server, i was searching a lot on the internet and i didn't get a sweetable answer to move on .. but finally i successfully made it ! .. It works with multiple domains, secure and no spam problems !

For the reader

I assume you have knowledge about how email works, Mail DNS records, linux, docker and SSL of course. This guide is not for beginners

If you dont have enough knowledge, take a seat and take a look here :

Term	Definition
Mail server	Is a computer system that sends and receives email source
	MX
CNAME	Used to alias one name to another,CNAME stands for Canonical Name. source
DKIM	Is an email authentication technique that allows the receiver to check that an email was indeed sent and authorized by the owner of that domain source
SPF	Is an email-authentication technique which is used to prevent spammers from sending messages on behalf of your domain source
DMARC	is an email validation system designed to protect your company’s email domain from being used for email spoofing source

Mailcow

Mailcow is a free, open source software project. A Mailcow server is a collection of Docker containers running different mail server applications, SOGo, Postfix, Dovecot etc. Mailcow provides a modern and easy to use web interface to create and manage email accounts. You can visit the official documentation

I ill not make a comparison about different opensource self hosted mail servers, rather then that i will give you a great repo that includes owesome softwares marked as self hosted, you can check the Email section come here

If you want some thoughts about mailcow from real users check this reddit discussions : come here and here

Server Requirements

The recommended OS to run mailcow is ubuntu 18.04, also don't use cento7 packages on cento8 because the maintainers said :

Do not use CentOS 8 with Centos 7 Docker packages. You may create an open relay.

For the resources i bought a VPS from OVH cloud provider with :

2 VCPU
4 GB of RAM
80 GB storage

Docker installed :



curl -sSL https://get.docker.com/ | CHANNEL=stable sh
systemctl enable docker.service
systemctl start docker.service

Docker-compose installed :



curl -L https://github.com/docker/compose/releases/download/$(curl -Ls https://www.servercow.de/docker-compose/latest.php)/docker-compose-$(uname -s)-$(uname -m) > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

If you have a firewall, you should allow those ports :



netstat -tulpn | grep -E -w '25|80|110|143|443|465|587|993|995|4190'

Note : you must have a clean server means no other applications or a reverse proxy becaue mailcow has every thing in place.

Okay now every thing is good ! but there is another thing to verify which is our IP !!!
we have to test it if it is blacklisted, if yes it's a huge problem ... we can't proceed anymore because your mails will not be delivered, it's all about your host reputation!.

We can check it with an online service called mxtoolbox by visiting this url mxtoolbox.

Basic DNS configuration

Before we proceed we have to get a domain name from any provider, i got one from OVH, and i delegated to AZURE DNS SERVICE, it dosen't matter actually, we will have the same configuration in any provider.

I pretend that i have :

Root Domain name : mymailserver.com
Server IP address : 1.2.3.4

Let's get started !

Create an A record

In your provider DNS pannel add an A record like this

Name : mail
Type : A
TTL : default
Value : 1.2.3.4

You can confirm with the dig command :



 dig mail.mymailserver.com +noall +answer

Create the CNAME records

In your provider DNS pannel add a CNAME record for the autoconfig like this

Name : autoconfig
Type : CNAME
TTL : default
Alias : mail.mymailserver.com

Test it with : dig autoconfig.mymailserver.com +noall +answer

Add another CNAME record for the autodiscover like this :

Name : autodiscover
Type : CNAME
TTL : default
Alias : mail.mymailserver.com

Test it with : dig autodiscover.mymailserver.com +noall +answer

Create the an MX record

In your root domain add an MX domain to point to your mail server domain :

Name : @/empty
Type : MX
TTL : default
Prefenerence/periorty : 10 or 0
Mail Exchange : mail.mymailserver.com

rDNS configuration

Get more information about rDNS here

You can configure your rDNS on the provider of your server and change the generated domain name to your mail domain mail.mymailserver.com

and test it with dig -x 1.2.3.4 +noall +answer, should get mail.mymailserver.com as responce.

Security DNS Records

Authenticate your mail server and protect it from Fake identities and Domain spoofing attacks we have to setup those records

SPF record

In your root domain add a TXT record like this :

Name : @/empty
Type : TXT
TTL : default
value : v=spf1 ip4:1.2.3.4 -all

Test it with dig mymailserver.com TXT

DKIM record

Setting up the dkim record needs a public key to be inserted in the record here but we can't now because we have to install mailcow and get the public key given by our mail server, we will leave it empty for now.

Name : dkim._domainkey
Type : TXT
TTL : default
value : v=DKIM1;k=rsa;t=s;s=email;p=

DMARC record

The best thing you can do for dmarc is to make a free account on dmarcian, it will give you a record like this :



v=DMARC1; p=reject; rua=mailto:<dmarc email1>; ruf=<dmarc email2>

Just add it to your domain DNS pannel like this :

Name : _dmarc
TTL : default
value : v=DMARC1; p=reject; rua=mailto: dmarc email1 ; ruf=dmarc email2

Test is with : dig _dmarc.mymailserver.com TXT

That's it for the DNS configurations, all records are in place execpt the dkim value we ill add it later

Install mailcow

You need git installed to clone the repo in your server :



 git clone https://github.com/mailcow/mailcow-dockerized
 cd mailcow-dockerized

Your in the repo now, so run the script ./generate_config.sh to generate your config, for the hostname add mail.mymailserver.com, it will request a certificate from Let'svEncrypt automatically.

To run your mail server just use this command :



docker-compose pull
docker-compose up -d

You need to wait some time to download all the containers, run them and provision a SSL certificate.

The default credentials to access are admin & moohoo (you must change it with your own)

Verify that all containers are healthy with the green icon like this :

Well done ! Let's proceed to add domains and mailboxes

Configure Mailcow

Our mail server is working now but there is neither domains or mailboxs configured. in this section we will go step by step .. let's GO !

Add your domain

Select configuration on the top bar then select mail setup
Click on add domain then add your root domain mymailserver.com and click on Add domain and restart SoGo
Select configuration again then click on configuration & details
In the hosizontal menu click configuration and click on ARC/DKIM keys
Scroll to bottom and fill the domain/s form with your domain mymailserver.com
On the selector type dkim
Select 2048 on the DKIM key length and click add
Copy the generated public key that starts with v=DKIM1;k=rsa;t=s;s=email;p= ...
Go to your DNS pannel and modify the TXT record with the generated value
Wait until your DNS modification propagate

You can validate your configuration for SPF, DKIM and DMARC with this site, you should get result like this (don't forget to put dkim as a selector` :

Create a Mailbox for mymailserver.com

By default mailcow give 10GB of storage to every domain and every mailbox under that domain has 3GB of storage .. so feel free to modify them to your needs,

To create a mailbox folow those steps :

Under configuration on the to bar click mail setup and select mailboxes`
Click add mailbox
Choose a username , your domain (it will be loaded automatically) and then add a password and click add
On the top bar click apps and then webmail, you will be redirected to the SoGo UI , login wiyh your newly created mail and password
on the bottom click on the green icon, you will get an interface of sending a mail
In your browser open another tab and go to this site : mail-tester.com and copy the mail address.
Return to your SoGo interface and send a mail with that mail, on the body make sure to add some text with a least two paragraphs.
Wait 10 seconds and go the mail-tester.com and click on then check your score

And BOOM you should get this result You Can Send :

Feel free to send a mail to your friends and your own Gmail address, don't worry Gmail still don't know your mail server, so he will placed as spam, just unspam it !, try to send it to zoho mail for example. The test again with other tools i suggest to use :

Mxtoolbox SuperTool and select test mail server
Dkimvalidator and send an email to the given address, make sure that all tests are oassed like the dmarcian.com tests.

Add another domain to mailcow

I assume we have a second domain : myseconddomain.com.
I will not repeat the same steps it's quite easy so let's go :

Create A record : mail.myseconddomain.com that points to your Host 1.2.3.4
Create a CNAME record : autoconfig that points to mail.myseconddomain.com
Create a CNAME record : autodiscover that points to mail.myseconddomain.com
Create an MX record: on the root domain add 10 as periority and mail.myseconddomain.com as target
Create SPF record like we did earlier
Create DMARC record as we did earlier
Create DKIM record and in the same time add your new domain as we did earlier and copy the generated DKIM key to your DKIM record.
Validate your records
Add a mailbox under your new domain and send an email to mail-tester.com and dkimvalidator.com, you should get 10/10 sweetheart :)

Final thoughts

Don't send a lot of mails directly you will be blocked ! ... so start step by step by sending 20 mails per day for the first week then try sending 80 the next week ... after 2 months you can send a lot of mails like 1000 mails, but take in mind that you need to add the list unsubscribe headers to your postfix to allow users to unsubscribe against your newsletters/subscriptions.

please follow me on twitter @hatem ben tayeb

Delivering React .. The hard way !

hatem ben tayeb — Thu, 31 Dec 2020 15:40:47 +0000

In this Post we will setup a React pipeline using Gitlab,Ansible and docker. we will go throught the whole process from nothing to a fast, reliable and hightly customizable pipeline with multi-environment deployment.

Daah ! let's start i can't wait 😍 !

😜

Tools

Before we start we need to define the damn 🔥 tech stack :

Gitlab : GitLab is a web-based DevOps lifecycle tool that provides a Git-repository manager providing wiki, issue-tracking and continuous integration and deployment pipeline features.
Ansible : Ansible is the simplest way to automate apps and IT infrastructure. Application Deployment + Configuration Management + Continuous Delivery.
Docker: Docker is a tool designed to make it easier to create, deploy, and run applications by using containers.

Note: if you dont know nothing about those tools ... No problem .... aaah actually it's a problem 😐 ... we are diving into advanced topic here 😝 ...

😕Come on ! i was kidding 😄

Actually no ...
contact me if you need some support

Architecture

yoo .. we have to draw the global environment architecture to get the whole picture about what we will do here 👌 ... dont start coding directly. Daaah 😤 ... you have to think by compiling the whole process in mind 😎

Of course we will create a repository ( i will not explain that 😏) on gitlab with a hello world react app ( i will not explain that 😏) and push it there.

Let's break down the architecture now :

Block 1 : this where our code application resides and the whole gitlab eco-system also, all configuration to start a pipeline must be there, actually you can install gitlab on your own servers .. but it's not the aim of this post.
Block 2: this is the important block for now (The CI environment) .. actually it is the server when all the dirty 💩 work resides like buiding docker containers .. saving cache ... testing code and so on ... we must configure this environment with love ❤️ haha yeah with love ... it's is the base of the pipeline speed and low level configurations.
Block 3 : the target environments where we will deploy our application using ansible playbooks via a secure tunnel .. SSH ... BTW i love you SSH 😄 because we will not install any runners on those targets servers we will interact with them only with ansible to ensure a clean deployment.

😙

CI environment

In this section we will connect our gitlab repo to the CI environment machine and install the gitlab runner on it of course.

Go to your repo ... under settings --> CI/CD --> runners and get the gitlab url and the token associeted to ... dont loose it 😑
You should have a VPS or a virtual machine on the cloud ... i will work on an azure virtual machine with ubuntu 18.04 installed
Install docker of course ... it's simple come here
Installing the gitlab runner :

curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_<arch>.deb"

dpkg -i gitlab-runner_<arch>.deb

Gitlab will be installed as service on your machine but i don't you can encounter a problem when starting it ... (don't ask me i don't know 😑 ) so you can start it as follow :

gitlab runner run & # it will work on background

You can now register the runner with gitlab-runner register and follow the instructions ... dont loose the token or reset it ... if you reset the token you have to re-register the runner again. i will make things easier ... here is my config.toml under /etc/gitlab-runner/config.toml

concurrent = 9 
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner-name"
  url = "https://gitlab.com/"
  token = "runner-token"
  executor = "docker"
  limit = 0
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    pull_policy = "if-not-present"
    tls_verify = false
    image = "alpine"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache:/cache"]
    shm_size = 0

let's make a breakdown here ...

This runner will run 9 concurent jobs on a docker containers (docker in docker) based on the alpine container (to make a clean build) ... The runner will pull new versions of images if they are not present ... This is optional you can turn it to always but we need to speed up the build ... No need to pull the same image again and again if there is no updates ... The runner will save the cache on the current machine under /cache on the host and pass it in use as a docker volume to save some minutes when gitlab by default upload the zipped cache to it's own storage and download it again ... It's painfull when the cache is becoming huge. At some point on time the cache will be so big .. So you can make your hand dirty and delete the shit 💩

We are almost done 😍

Now you can go the repository under settings --> CI/CD --> runners and verify that the runner was registred successfully ( the green icon )

. . .

The react pipeline

let's code the pipeline now 😄 .... wait a second !!! we need the architecture as the previous section ... so here is how the pipeline will look like ...

This pipeline aims to support the folowing features :

Caching node modules for faster build
Docker for shiping containers
Gitlab private registry linked to the repo
Ship only /build on the container with nginx web server
Tag containers with the git SHA-COMMIT
Deploy containers with an ansible playbook
SSH configuration as a gitlab secret to secure the target IP
Only ssh keypairs used for authentication with the target server ... no damn passwords 💩 ...

. . .

Defining Secrets

This pipeline needs some variables to be placed in gitlab as secrets on settings --> CI/CD --> Variables :

Variable name	Role	Type
ANSIBLE_KEY	The target server ssh private key 😐	file
GITLAB_REGISTRY_PASS	Gitlab registry password (your account password 😐)	variable
GITLAB_REGISTRY_USER	Gitlab registry login (your account user 😐)	variable
SSH_CFG	The regular ssh config that contains the target IP	file

The SSH_CFG looks like this :

Host *
   StrictHostKeyChecking no

Host dev 
    HostName <IP>
    IdentityFile ./keys/keyfile
    User root

Host staging 
    HostName <IP>
    IdentityFile ./keys/keyfile
    User root

Host prod 
    HostName <IP>
    IdentityFile ./keys/keyfile
    User root

I will not explain this 😭 ... come here

. . .

[KNOCK KNOCK](https://bongo.cat/) ... are you still here 😺

Thank god 😄 ! his here 👶 ... let's continue then be ready 🔥 ...

. . .

Preparing Dockerfile

Before writing the dockerfile take in mind that the steup should be compatible with the pipeline architecture ... if you remember we have a separate jobs for :

Installing node modules
Run the build process

So the Dockerfile must contain only the builded assets only to be served by nginx 😄

Here is our sweet 🐭 Dockerfile :

FROM nginx:1.16.0-alpine
COPY build/  /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d
RUN mv  /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.old
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

This dockerfile does not do too much work, it just take the /build directory and copy it under /usr/share/nginx/html to be served.

Also we need a basic nginx config like follow to be under /etc/nginx/conf.d:

server {
  include mime.types;
  listen 80;
  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
    try_files $uri $uri/ /index.html;
  }
  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
    root   /usr/share/nginx/html;
  }
}

You see ! 👀 its simple let's proceed to setup the ansible playbook for the deployment process ... hurry up 😐

. . .

Deployment with ansible

We are almost done ! the task now is to write the ansible playbook that will do the folowing :

Create a docker network and specify the the gateway address
Authenticate the gitlab registry
Start the container with the suitable configurations
Clean the unsed containers and volumes
Most setup will be in the inventory file

Let's take a look at the inventory_file:

[dev]
devserver ansible_ssh_host=dev ansible_ssh_user=root ansible_python_interpreter=/usr/bin/python

[dev:vars]
c_name={{ lookup('env','CI_PROJECT_NAME') }}-dev #container name
h_name={{ lookup('env','CI_PROJECT_NAME') }}-dev #host name
subnet=172.30.0  # network gateway                                         
network_name=project_name_dev
registry_url={{ lookup('env','CI_REGISTRY') }}                          
registry_user={{ lookup('env','GITLAB_REGISTRY_USER') }}    
registry_password={{ lookup('env','GITLAB_REGISTRY_PASS') }}  
image_name={{ lookup('env','CI_REGISTRY_IMAGE') }}:{{ lookup('env','CI_COMMIT_SHORT_SHA') }}-dev 

[project_network:children]
dev
[project_clean:children]
dev

The ansible_ssh_host=dev refers to the SSH_CFG configuration.

Gitlab by default exports many useful environment variables like :

CI_PROJECT_NAME : the repo name
CI_COMMIT_SHORT_SHA : the sha commit ID to tag the container

You can explore all variables here.

Let's move now to the playbook ... i'm tired damn it haha .. it is a long post ... okay nevermind come on ..

Here is the ansible playbook :

---
- hosts: project_network
  #become: yes # for previlged user
  #become_method: sudo   # for previlged user
  tasks:                                                     
  - name: Create docker network
    docker_network:
      name: "{{ network_name }}"
      ipam_config:
        - subnet: "{{ subnet }}.0/16"
          gateway: "{{ subnet }}.1"

- hosts: dev
  gather_facts: no
  #become: yes # for previlged user
  #become_method: sudo   # for previlged user
  tasks:

  - name: Log into gitlab registry and force re-authorization
    docker_login:
      registry: "{{ registry_url }}"
      username: "{{ registry_user }}"
      password: "{{ registry_password }}"
      reauthorize: yes

  - name : start the container
    docker_container:
      name: "{{ c_name }}"
      image : "{{ image_name }}"
      pull: yes
      restart_policy: always
      hostname: "{{ h_name }}"
      # volumes:
      #   - /some/path:/some/path
      exposed_ports:
        - "80"
      networks:
        - name: "{{ network_name }}"
          ipv4_address: "{{ subnet }}.2"
      purge_networks: yes

- hosts : project_clean
  #become: yes # for previlged user
  #become_method: sudo   # for previlged user
  gather_facts : no 
  tasks: 

  - name: Removing exited containers
    shell: docker ps -a -q -f status=exited | xargs --no-run-if-empty docker rm --volumes
  - name: Removing untagged images
    shell: docker images | awk '/^<none>/ { print $3 }' | xargs --no-run-if-empty docker rmi -f
  - name: Removing volume directories
    shell: docker volume ls -q --filter="dangling=true" | xargs --no-run-if-empty docker volume rm

This playbook is a life saver because we configure the container automatically before starting it ... no setup on the remote host ... we can deploy the same in any other servers based on linux. the container update is quite simple .. ansible will take care of stopping the container and starting new one with different tag and then clean up the shit 💩

We can also make a rollback to the previous container by going to the previous pipeline history on gitlab and restart the lastest job the deploy job because we have already an existing container on the registry 😄

The setup is for dev environment you can copy paste the two files for the prod & staging environment ...

. . .

Setting up the Pipeline

The pipeline will deploy to the three environments as i mentioned on the top of this post ...

Here is the full pipeline code :


variables: 
  DOCKER_IMAGE_PRODUCTION : $CI_REGISTRY_IMAGE 
  DOCKER_IMAGE_TEST : $CI_REGISTRY_IMAGE   
  DOCKER_IMAGE_DEV : $CI_REGISTRY_IMAGE


#caching node_modules folder for later use  
.example_cache: &example_cache
  cache:
    paths:
      - node_modules/


stages :
  - prep
  - build_dev
  - push_registry_dev
  - deploy_dev
  - build_test
  - push_registry_test
  - deploy_test
  - build_production
  - push_registry_production
  - deploy_production


########################################################
##                                                                                                                                 ##
##     Development: autorun after a push/merge                                               ## 
##                                                                                                                                 ##
########################################################

install_dependencies:
  image: node:12.2.0-alpine
  stage: prep
  <<: *example_cache
  script:
    - npm ci --log-level=error 

  artifacts:
    paths:
      - node_modules/  
  tags :
    -  runner_name 
  only:
    refs:
      - prod_branch
      - staging_branch
      - dev_branch
    changes :
      - "*.json"

build_react_dev:
  image: node:12.2.0-alpine
  stage: build_dev
  <<: *example_cache
  variables:
    CI : "false"
  script:
    - cat .env.dev > .env
    - npm run build

  artifacts:
    paths:
      - build/
  tags :
    -  runner_name 
  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "dev_branch"'


build_image_dev:
  stage: push_registry_dev
  image : docker:19
  services:
    - docker:19-dind
  variables: 
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  before_script:
  # docker login asks for the password to be passed through stdin for security
  # we use $CI_JOB_TOKEN here which is a special token provided by GitLab
    - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
  script:
    - docker build  --tag $DOCKER_IMAGE_DEV:$CI_COMMIT_SHORT_SHA-dev .
    - docker push $DOCKER_IMAGE_DEV:$CI_COMMIT_SHORT_SHA-dev
  tags :
    - runner_name
  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "dev_branch"'


deploy_dev:
  stage: deploy_dev
  image: willhallonline/ansible:latest
  script:
    - cat ${SSH_CFG} > "$CI_PROJECT_DIR/ssh.cfg"               
    - mkdir -p "$CI_PROJECT_DIR/keys"                            
    - cat ${ANSIBLE_KEY} > "$CI_PROJECT_DIR/keys/keyfile"      
    - chmod og-rwx "$CI_PROJECT_DIR/keys/keyfile"    
    - cd $CI_PROJECT_DIR && ansible-playbook  -i deployment/inventory_dev --ssh-extra-args="-F $CI_PROJECT_DIR/ssh.cfg -o ControlMaster=auto -o ControlPersist=30m" deployment/deploy_container_dev.yml
  after_script:
    - rm -r "$CI_PROJECT_DIR/keys" || true                              
    - rm "$CI_PROJECT_DIR/ssh.cfg" || true

  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "branch_dev"'
  tags :
    - runner_name

########################################################
##                                                                                                                                 ##
##     pre-production: autorun after a push/merge                                            ## 
##                                                                                                                                 ##
########################################################

build_react_test:
  image: node:12.2.0-alpine
  stage: build_test
  <<: *example_cache
  variables:
    CI : "false"
  script:
    - cat .env.test > .env
    - npm run build

  artifacts:
    paths:
      - build/
  tags :
    -  runner_name 

  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "staging_branch"'


build_image_test:
  stage: push_registry_test
  image : docker:19
  services:
    - docker:19-dind
  variables: 
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  before_script:
  # docker login asks for the password to be passed through stdin for security
  # we use $CI_JOB_TOKEN here which is a special token provided by GitLab
    - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
  script:
    - docker build  --tag $DOCKER_IMAGE_TEST:$CI_COMMIT_SHORT_SHA-test .
    - docker push $DOCKER_IMAGE_TEST:$CI_COMMIT_SHORT_SHA-test

  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger" && $CI_COMMIT_BRANCH == "staging_branch"'
  tags :
    - runner_name



deploy_test:
  stage: deploy_test
  image: willhallonline/ansible:latest
  script:
    - cat ${SSH_CFG} > "$CI_PROJECT_DIR/ssh.cfg"               
    - mkdir -p "$CI_PROJECT_DIR/keys"                            
    - cat ${ANSIBLE_KEY} > "$CI_PROJECT_DIR/keys/keyfile"      
    - chmod og-rwx "$CI_PROJECT_DIR/keys/keyfile"    
    - cd $CI_PROJECT_DIR && ansible-playbook  -i deployment/inventory_test --ssh-extra-args="-F $CI_PROJECT_DIR/ssh.cfg -o ControlMaster=auto -o ControlPersist=30m" deployment/deploy_container_test.yml
  after_script:
    - rm -r "$CI_PROJECT_DIR/keys" || true                              
    - rm "$CI_PROJECT_DIR/ssh.cfg" || true
  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger" && $CI_COMMIT_BRANCH == "staging_branch"'
  tags :
    - runner_name

########################################################
##                                                                                                                                 ##
##     Production: must be deployed manually                                                    ## 
##                                                                                                                                 ##
########################################################

build_react_production:
  image: node:12.2.0-alpine
  stage: build_production
  <<: *example_cache
  variables:
    CI : "false"
  script:
    - cat .env.prod > .env
    - npm run build

  artifacts:
    paths:
      - build/
  tags :
    -  runner_name 
  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "prod_branch"'
      when: manual

build_image_production:
  stage: push_registry_production
  image : docker:19
  services:
    - docker:19-dind
  variables: 
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  before_script:
  # docker login asks for the password to be passed through stdin for security
  # we use $CI_JOB_TOKEN here which is a special token provided by GitLab
    - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
  script:
    - docker build  --tag $DOCKER_IMAGE_PRODUCTION:$CI_COMMIT_SHORT_SHA .
    - docker push $DOCKER_IMAGE_PRODUCTION:$CI_COMMIT_SHORT_SHA

  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "prod_branch"'
  tags :
    - runner_name
  needs: [build_react_production]



deploy_production:
  stage: deploy_production
  image: willhallonline/ansible:latest
  script:
    - cat ${SSH_CFG} > "$CI_PROJECT_DIR/ssh.cfg"               
    - mkdir -p "$CI_PROJECT_DIR/keys"                            
    - cat ${ANSIBLE_KEY} > "$CI_PROJECT_DIR/keys/keyfile"      
    - chmod og-rwx "$CI_PROJECT_DIR/keys/keyfile"    
    - cd $CI_PROJECT_DIR && ansible-playbook  -i deployment/inventory --ssh-extra-args="-F $CI_PROJECT_DIR/ssh.cfg -o ControlMaster=auto -o ControlPersist=30m" deployment/deploy_container.yml
  after_script:
    - rm -r "$CI_PROJECT_DIR/keys" || true                              
    - rm "$CI_PROJECT_DIR/ssh.cfg" || true
  rules:
    - if: '$CI_PIPELINE_SOURCE != "trigger"  && $CI_COMMIT_BRANCH == "prod_branch"'
  tags :
    - runner_name
  needs: [build_image_production]

Here is some notes about this pipeline:

The pipeline is protected by default to not be started with the trigger token ( Gitlab pipeline trigger)
The prep stage will start if there is any modifications in any json file including the package.json file
The pipeline jobs runs on docker alpine image (DinD) so we need some variables to connect to the docker host by using DOCKER_HOST: tcp://docker:2375/ and DOCKER_TLS_CERTDIR: ""
The production deployment depends on the staging jobs to be succeeded and tested by the testing team. by default no auto deploy to prod ... it's manual !
I used some files to store application environment variables using .env.dev , env.test and .env.prod you can use what you want !
Make sure to use a good docker image for the job based images .. for node i always work with LTS versions.
Create a deployment folder to store the ansible playbooks and inventory files.
Create a Cron Job to delete the cache every three months to clean the cache on the CI environment.
On the target server make sure to install docker, nginx, certbot and docker python package

**. . .**

Final thoughts

You can make this pipeline as template to deliver other kinds of projects like :

Python
Rust
Node
Go

I hope this post was helpful ! thanks for reading 😄 it was great to share this with you, if you have any problems in setting this just let me know !

Thanks 😍

Optimizing CI/CD Pipeline for Rust Projects (Gitlab & Docker)

hatem ben tayeb — Sun, 27 Dec 2020 20:42:33 +0000

What is Rust ?

Rust is a programming language ( general purpose) C-like, which mean it is a compiled language and it comes with new strong features in managing memory and more. The cool thing ! rust does not have a garbage collector and that is awesome 😅 .

What is DevOps ?

In short, Devops is the key feature that helps the dev team and the ops team to be friends 😃 without a work conflicts , It is the ART of automation. It increase the velocity of delivering a better softwares !

Identifying the problem

we can make a lot of things with rust like web apps , system drivers ans much more but there is one problem which is the time that rust takes to make a binary by downloading dependencies and compile them.

The cargo command helps us to download packages ( crates in the rust world) , The Rustc is our compiler. Now we need to make a pipeline using the Gitlab CI/CD and docker to make the deployment faster.

This is our challenge and the Goal of this article ! 👊

Static linking Vs Dynamic linking

Rust by default uses a Dynamic linking method to build the binary, so what is dynamic linking ?.

The Dynamic linking uses shared libraries , so the lib is loaded into the memory and only the address is integrated into the binary. In this case the libc is used.

The Static linking uses static libraries which is integrated physically into the binary, no addresses are used and the binary size will be more bigger. In this case the musl libc is used.

You want to know more ? Then check this : click here.

Optimizing the CI/CD pipeline

The CI/CD pipeline is a set a steps that allow us to make :

build → test → deploy

In this article i will focus on the build stage because in my opinion it is very sensitive phase and it will affect the “Time to market” approach !

So the first thing is to optimize the size of our docker images to make the deployment faster. Before we begin, i will use a simple rust project for the demo.

let’s understand the project structure :

src : This dir contains all source code of the app (*.rs files).
Cargo.toml : This file contain the package meta-data and the dependencies required by the app and some other features … .
Cargo.lock : Ct contains the exact information about your dependencies.
Rocket.toml : With this file we specify the app status ( development , staging or production) and the required configuration for each mode, for example the port configuration for each environment.
Dockerfile : This is the docker file configuration to build the image with the specific environment that is configured already in Rocket.toml.

Are you prepared 👊 😈 !!! , let’s begin the show !! 🎉 🎉 🎉

We will begin by building the app image locally , so let’s see how the docker file look like :

FROM rustdocker/rust:nightly as cargo-build 
RUN apt-get update 
RUN apt-get install musl-tools -y 
RUN /root/.cargo/bin/rustup target add x86_64-unknown-linux-musl 
RUN USER=root /root/.cargo/bin/cargo new --bin material 
WORKDIR /material 
COPY ./Cargo.toml ./Cargo.toml 
COPY ./Cargo.lock ./Cargo.lock 
RUN RUSTFLAGS=-Clinker=musl-gcc /root/.cargo/bin/cargo build --release --target=x86_64-unknown-linux-musl --features vendored 
RUN rm -f target/x86_64-unknown-linux-musl/release/deps/material* 
RUN rm src/*.rs 
COPY ./src ./src 
RUN RUSTFLAGS=-Clinker=musl-gcc /root/.cargo/bin/cargo build --release --target=x86_64-unknown-linux-musl --features vendored 
FROM alpine:latest 
COPY --from=cargo-build /auth/target/x86_64-unknown-linux-musl/release/material . 
CMD ["./material"]

This Dockerfile is splitted into two sections :

The builder section ( a temporary container)
The final image (Reduced in size)

The builder section:
In order to use rust we have to get a pre-configured images that contains the Rustc compiler and the Cargo tool. the image have the rust nightly build version and this is a real challenge because it’s not stable 😠.

We will use the static linking to get fully functional binary that doesn’t need any shared libraries from the host image !!

let’s breakdown the code :

First we import the base image.
We need the MUSL support : musl-tool after updating the source.list of your packages apt-get update , MUSL is an easy-to-deploy static and minimal dynamically linked programs.
Now we have to specify the target , if you don’t know ! no problem ! you can use x86_64-unknown-linux-musl , run with Rustup (the rust toolchain installer)
To define the project structure on the container we use cargo new --bin material (material is the project name), it’s much like the structure that we see earlier.
Making the material directory as a default we use the WORKDIR Dockerfile command.
The Cargo.toml and Cargo.lock are required for deps. installation
Setting up the RUST_FLAGS with -Clinker=musl-gcc : this flag tell cargo to use the musl gcc to compile the source code , the --release argument is used to prepare the code for a release ( final binary optimization).
--target specify the target compilation 64 or 32 bit
--feature vendored thsi command is an angle 😄 ! it helps to solve any ssl problem by finding the SSL resources automatically without specifying the SSL lib directory and the SSL include directory. It saves me a lot of time, this command is associated with some configurations in the Cargo.toml file under the feature section.

Until now we only build the dependencies in Cargo.toml and we make the clean ( removing unnecessary files)

After downloading and compiling required packages, it’s the time to get the source code into the container and make the final build to produce the final binary ( standalone).

The builder stage has complete ! congrats 😙 🎉 yeah !!. Now let’s use alpine as a base image to get the binary from the build stage , but ! wait a second ! what is alpine ???

Alpine is a Linux distribution, it’s characterized in the docker world by his size ! it is a very small image (4MB) and it contains only the base commands (busybox)

--from=cargo-build ..../material now we will copy the final binary to the alpine and the intermediate container (cargo-build) will be destroyed and we get as a result a very tiny image (12–20MB) ready to use 😃 😃 😃

You know how to build a docker image right 😲 ? okay 😃

The CI/CD pipeline

After testing the image locally, it seems good 😃, we resolve the docker image size, but in CI system the velocity is very important than size !! so let’s take this challenge and reduce the compilation time of this rust project !!

let’s look at the .gitlab-ci.yml file ( our CI configuration):

.caching_rust: &caching_rust
    cache:
      paths:
        - .cargo/
        - .cache/sccache
        - target/x86_64-unknown-linux-musl/release/material

stages:
    - build_binary
    - build_docker


prepare_deps_for_cargo:
   stage: build_binary
   image: hatembt/rust-ci:latest
   <<: *caching_rust
   before_script:
       - export CARGO_HOME="${PWD}/.cargo"
       - echo $CARGO_HOME
       - export SCCACHE_DIR="${PWD}/.cache/sccache"
       - echo $SCCACHE_DIR
       - export PATH="/builds/Astrolab-devops/material/.cargo/bin:$PATH"
       - export RUSTC_WRAPPER="$CARGO_HOME/bin/sccache"
       - echo $RUSTC_WRAPPER

   script:
       -  cargo build --release --target=x86_64-unknown-linux-musl --features vendored
   cache:
     paths:
       - .cargo/
       - .cache/sccache
   artifacts:
     paths:
       - target/x86_64-unknown-linux-musl/release/material


build_docker_image:
   stage: build_docker
   image: docker:latest
   << : *caching_rust
   services:
     - docker:dind
   script:
     - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
     - docker build -t registry.gitlab.com/astrolab-devops/material:0.2.1 .  
     - docker push registry.gitlab.com/astrolab-devops/material:0.2.1

There is a tip in this file , i just splitted the docker file into two stages in this .gitlab-ci.yml :

The builder stage (rustdocker/rust..)→ build dependencies and binary
The final stage (Alpine) → the build stage

For the CI work i prepared a ready-to-use docker image that contains all i need to make a reliable and fast pipeline for rust project , this image is hosted in my $docker hub .

hatembt/rust-ci:latest

This image contains the following packages installed and configured :

The sccache command : this command caches the compiled dependencies ! so by making this action to our build we can compile deps only one time !! 😅 , and we gained much more time.
The cargo-audit : it’s a helpful command let’s us to scan dependencies security.

Let’s breakdown the code and understand what’s going on !!

In the first job : prepare_deps_for_cargo we need our base image hatembt/rust-ci .

In this job some setting are required to make a successful build are placed in the before_script:

Defining the cargo home in the path variable.
Defining the cache directory that s generated by sccache (it contains the compilation cache ).
Adding cargo and rustup ( they are under .cargo/bin) in the path.
Specifying the RUSTC_WRAPPER variable in order to use the sccache command with the rustc or MUSL in our case.

Now all thing are ready ! so let’s make the build in the script section, you are already now what we should do 😃 , let’s skip it 👇.

The cache and artifacts sections are very important ! its saves the data under :

.cargo/
.cache/sccache
target/x86_64-unknown-linux-musl/release/material (this is our final binary ).

To know more about caching and artifacts flow this link.

All data that is created in the first run of the CI jobs will be now saved and uploaded to the Gitlab coordinator. On the next build (new codes are pushed), we will not start the build from scratch, we just build the new packages , the old data will be injected with <<:*caching_rust after the image keyword.

let’s move on the next JOB : build_docker_image:

I made a new Dockerfile for the docker build stage, it’s based on the alpine image and it contain only the binary from the previous stage.

The new Dockerfile:

FROM alpine:latest
COPY target/x86_64-unknown-linux-musl/release/material .
CMD ["./material"]

First we need a docker in docker image (dind) → to get the docker command and let’s make the steps below:

Login to the Gitlab registry
Build the image with the new Dockerfile
Push the image to Gitlab registry

And Now the results ! 😧

The image size is :

The CI Time :

NB: the time is for the hole build time , the build binary and docker_build stages.

This is the power of Devops, the art of automation with some philosophy in the configurations and the steps to flow we can make even better than these results.

In business the velocity ,the quality and the necessary features (on the application) are very important to Bring the company on the hight levels of success → this is the successful Digital transformation.

Finally, i hope that this Story helps you to move on to next steps in the CI/CD systems, you can apply these ideas into any language (mostly complied languages, but still the same steps). If you have any feedback or critiques, please feel free to share them with me.

Thank you 😄

Deploying a Dockerized Angular App with Github Actions

hatem ben tayeb — Sat, 26 Dec 2020 19:41:02 +0000

In this article we will discover the devops movement step by step, we will talk about the fundamental concepts then we will make a basic pipeline with github actions to deploy an angular 6 app, so let’s go.

What is devops ?

Devops is used to remove the conflict between the developers team and the operations team to work together. This conflict is removed by adding a set of best practices, rules and tools. The devops workflow is defined with a set of setps :

Plan

This is the first step, where the team defines the product goals and phases, also defining deadlines and assigning tasks to every team member, this step is the root of the hole workflow. the team uses many methodology like scrum and agile.

Code:

After planning, there is the code when the team converts the ideas to code. every task must be coded and merged to the main app, here we use an SCM to organize the collaboration to make a clean code and have a full code history to make a rollback in case of failure.

Build:

After coding we push the code to Github or Gitlab ( SCM ) and we make the build, usually we use docker images for packaging. also we can build the code to be a Linux package like deb , rpm … or even zip files , also there is a set of tests like unit tests and integration tests. this phase is critical !

Test:

The build was succeeded, no it’s time to deploy the build artifacts to the staging server when we apply a set of manual and automated tests ( UAT ).

Release:

it’s the final step for the code work, so we make a release and announce a stable version of our code that is fully functional ! also we can tag it with a version number .

Deploy:

A pre-prod or a production server is the target now, to make our app up and running

Operate:

It’s all about infrastructure preparation and environment setup with some tools like terraform for IaaC, ansible for configuration management and security stuff configurations …

Monitor:

The performance is very important, so we install and configure some monitoring tools like ELK, nagios and datadog to get all information about the applications like CPU and memory usage …

Deploying an angular app

In this example we will deploy a simple angular app on two environments.

On VPS ( OVH provider) as a development server.
on heroku as a staging server.

So you must have a VPS and a heroku account to continue with me.
The application repository is here : Github repo.

Clone the project with git clone https://github.com/hatembentayeb/angular-devops
run npm install && ng serve to run the app locally

Preparing the deployment for heroku

Nginx is a popular and powerful web server can be used to serve a large variety of apps based on python, angular and react …

I will go through an optimization process to produce a clean and a lightweight docker container with the best practices as much as i can.

Writing the Dockerfile

First we will prepare the Dockerfile to be deployed to the heroku cloud,so there is some tricks to make it work smoothly, make sure that you have an account and simply click new to create an app :

Make sure to give a valid name for your app, then go to your accout settings and get your API_KEY that we will use it in the pipeline file:

let’s take a look at the dockerfile of the app:

FROM trion/ng-cli as builder
WORKDIR /app
COPY package.json package.json
COPY package-lock.json package-lock.json
RUN npm ci  --debug 
COPY . .
RUN ng build --prod

FROM nginx:1.17.5
COPY default.conf.template /etc/nginx/conf.d/default.conf.template
COPY nginx.conf /etc/nginx/nginx.conf
COPY --from=builder  /app/dist/my-first-app /usr/share/nginx/html 
CMD /bin/bash -c "envsubst '\$PORT' < /etc/nginx/conf.d/default.conf.template > /etc/nginx/conf.d/default.conf" && nginx -g 'daemon off;'

This Dockerfile is splitted into two stages :

Builder stage : The name of the stage is builder, it is a temporary docker container that produces an artifact which is the dist/ folder created by ng build --prod that compiles our project to produce a single html page and some *js & *.css . The base images that is used here is trion/ng-cli that containes all requirements to run an angular up and it’s accessible for public use in the Docker-hub, the public docker registry.
Make sure to install all app requirement packages with npm ci , the ci command is used often in the continues integration environments because it is faster than npm install.
Final stage: The base image for this stage is nginx:1.17.5 and simply we copy the dist/ folder from the builder stage to the /var/share/nginx/html folder in the nginx container with the command COPY --from=builder ...
There is additional configurations required to run the app, we need to configure nginx, there is a file named default.conf.template that contains a basic nginx configurations so we copy it to the container under /etc/nginx/conf.d/default.conf.template , this file have the $PORT variable that have to be changed when building the docker image in the heroku environment.
The default.conf.template :

server {                         
listen $PORT default_server;

location / {                           
include  /etc/nginx/mime.types;                                                      
root   /usr/share/nginx/html/;
index  index.html index.htm;       
}                                                                      
}

Also make sure to copy the nginx.conf under the /etc/nginx/nginx.conf , you are free to change and modify 😃, but for now i will use the default settings.
The last command is a little bit confusing so let’s break it down :

CMD /bin/bash -c “envsubst ‘\$PORT’ < /etc/nginx/conf.d/default.conf.template > /etc/nginx/conf.d/default.conf” && nginx -g ‘daemon off;’

→ /bin/bash -c ‘ command’ : This command will run a linux command with the bash shell.
→ envsubst : It is a program substitutes the values of environment variables, so it will replace the $PORT from the heroku environment and replace it in the default.conf.template file with it’s value, this variable is given by heroku and attached to your app name, then we rename the template with default.conf which is recognized by nginx.
→ nginx -g ‘daemon off;’: The daemon off; directive tells Nginx to stay in the foreground. For containers this is useful as best practice is for one container = one process. One server (container) has only one service.

Preparing the deployment for the VPS on OVH

We will use the VPS as a development server so no need for a docker now we will use ssh for this, after all make sure to have a VPS , ssh credentials and a public IP.

I assume you have nginx installed , if not try to do it, it is simple 😙

In this tutorial i will be using the sshpass command, it is powerful and suitable for CI environments.

You can install it with : apt-get install sshpass -y .

lets deploy the app to our server from the local machine, navigate to the repo and run ng build --prod , then navigate to dist/my-first-app folder and type this command :

sshpass  scp -v -p <password>  -o stricthostkeychecking=no -r *.* root@<vps-ip>:/usr/share/nginx/html

If you don’t want to hardcode the password in the command line try to set the SSHPASS variable with your password like this export SSHPASS="password"and replace -p with -e to use the environment variable.

Now all things almost done ! great 😃 ! let’s prepare the pipeline in the github actions which is a fast and powerful ci system provided by github inc.

Under the project root folder create the file main.yml in the github/wokflows folder, this directory is hidden so must start with a point like this : .github/workflows/main.yml

Preparing the pipeline

let’s take a look at the pipeline steps and configurations :


name: Build an angular project 
on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [12.x]

    steps:
      - uses: actions/checkout@v1
      - name: Cache node modules
        uses: actions/cache@v1
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
      - name: Node ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}

      - name : push to staging server with ssh
        env :
            SSHPASS : ${{ secrets.SSHPASS }}
            SERVER : ${{ secrets.SERVER}}
        run : |
            sudo apt-get update 
            sudo apt-get install sshpass -y
            npm install -g @angular/cli@6.2
            npm ci --debug
            ng build --prod
            cd dist/my-first-app/
            sudo sshpass  -p ${SSHPASS}   -v  ssh -o StrictHostKeyChecking=no root@${SERVER} 'rm -rf /usr/share/nginx/html/*'
            sudo sshpass -p ${SSHPASS} scp -v  -o stricthostkeychecking=no -r *.* root@${SERVER}:/usr/share/nginx/html

      - name: push to heroku
        env : 
            HEROKU_REGISTRY_URL: registry.heroku.com
            HEROKU_TOKEN : ${{ secrets.HEROKU_TOKEN }}
        run : |
            docker login --username=_ --password $HEROKU_TOKEN $HEROKU_REGISTRY_URL
            export HEROKU_API_KEY=$HEROKU_TOKEN
            heroku container:login
            heroku container:push web --app angulardevops
            heroku container:release web --app angulardevops
            heroku ps:scale web=1 --app angulardevops

Block 1: In this block we define the the workflow name and the actions that must be performed to start the build , test and the deployment. and of course you have to specify the branch of your repo (by default master ).
Block 2 : The jobs keyword has to sub keywords build and steps , the build define the base os for the continues integration environment, in this case we will use ubuntu-latest , also we define the node-version as a matrix that allow us to use multiple node versions in the list, in this case we need only 12.x . The steps allow us to define the wokflow steps and configurations ( build,test,deploy...).
Block 3 : actions/checkout@v1 is used to clone the app code in the ci env. this action is provided by github. Lets define a cache action with the name cache node modules , the name is up to you 😃, then we use a predefined action called actions/cache@v1 and specify the folders that we want to cache.
Block 4 : Installing and configuring the node run-time with an action called actions/node-setup@v1 and pass to it the desired node version that we already defined.
Block 5 : The show will begin now ! let’s configure the build and the deployment to the VPS. Create two environment variables SSHPASS for the sshpass command and define the server address , make sure to define these values on the github secrets under setting on the top of your repo files. Under run keyword put your deployment logic. so we need the sshpass command and and the angular cli to be installed, then install all required packages and build the app with the production mode --prod , next, navigate to the dist/my-first-app folder and run the sshpass command with a set of arguments to remove older app in the server and deploy the new code.
Block 6 : Now heroku is our target, so define also two env. variables, the heroku registry url and the API KEY to gain access to the registry using docker , next we need to define a special variable HEROKU_API_KEY that is used by heroku cli, next, we login to the heroku container and build the docker image then we pushed to the registry. we need to specify the target app in my case i named it angulardevops . After deploying the docker image we need to release it and tell the heroku dynos to run our app on a heroku server, using 1 server web=1 , note that web is the name of the docker image that we already pushed.

We are almost done! now try to make a change in the app code and push it to GitHub , the workflow will start automatically 🎉 🎉 😄 !

You can view the app here: https://angulardevops.herokuapp.com/

Finally, this tutorial is aimed to help developers and DevOps engineers to deploy an Angular app, I hope it is helpful 😍. for any feedback please contact me!

If this post was helpful click the clap button as much as possible 😃.

Thank you 😃

Hello Docker 😄

hatem ben tayeb — Fri, 25 Dec 2020 18:58:54 +0000

In this story i will share my docker experience, so, be ready ! to know the most used container technologies in the hole world ! . I will help you to get started in docker also we will deep dive into cool features … , and by the end of this story you will be able to make your own projects using docker

Common Docker commands

In this section I am going to show you the most used docker commands so let’s begin :

To get info about your docker environment use : docker info
To remove a container use : docker rm <name | ID>
To remove an entire image use : docker rmi <name | ID>
To remove a container after run use : docker run --rm <name|ID>
To view current running containers use : docker ps
To view all running and exited containers use: docker ps -a
To view all and only the IDs of containers use : docker ps -a -q
To get all images use : docker images
To remove all images use : docker rm $(docker ps -a -q)
To view the images use : docker images -f"dangling=true"
To remove them use : docker rmi $(docker images -f"dangling=true" -q)
To run the container in background use : docker run -d ... <name|ID> That’s enough for now 😆, no lets build a simple python project with flask

The Flask app

Flask is a simple and powerful Framework for python like apache or tomcat... So let's begin :

app.py

from flask import Flask
app = Flask(__name__)

@app.route("/")
    def hello():
        return "Hello My Name is Hatem"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Make sure to install flask with pip install flask and run the script with python app.py and open your browser on localhost:8080/ .

Let’s generate the requirements.txt , we will not use pip freeze here (we are not in a virtual environment) but we will use pipreqs and make sure to install it with pip install pipreqs .
use pipreqs <path to the python project> to generate it.
Now let’s write the Dockerfile :

Dockerfile

FROM python:latest
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY app.py .
CMD python app.py

FROM python:latest : using python as a base image.
WORKDIR /app : using /app as a default directory.
COPY requirements.txt . copy this file to /app
RUN pip install -r req..txt : install the required dependencies
COPY app.py . : copy the app script to /app
CMD python app.py : The container entry point when we run it

Run the build process with docker build -t hello_flask .
Run the container docker rn --rm -p 8080: 8080 hello_flask , the -p 8080:8080 : will map the port 8080 on your host to the port 8080 in your container → this is the port mapping in docker, you can change the host port to any port you want and must be > 1024 ( ports < 1024 are reserved by the system).

Environment variables in Docker

An environment variable is a variable whose value is set outside the program, typically through the functionality built into the operating system or microservice. An environment variable is made up of a name/value pair, and any number may be created and available for reference at a point in time.
→ source for more information.
let’s use them in our example :

import os 
...
@app.route("/")
def hello():
    return "Hello My Name is {}".format(os.environ['NAME'])

the os.environ['NAME'] will fetch the NAME variable and get its value and return it in the browser.
make sure to build the container again and run it like this :
docker run --rm -p 8080:8080 -e "NAME=steve" hello_flask

Volumes with Docker

In order to be able to save (persist) data and also to share data between containers, Docker came up with the concept of volumes. Quite simply, volumes are directories (or files) that are outside of the default Union File System and exist as normal directories and files on the host filesystem.
→ source
let’s make some change to our script :

@app.route("/name_from_file")
def name_from_file():
    with open("files/name.txt","r") as file :
        name= file.readline()
    return "Hello My Name is {}".format(name)

Make sure to add a directory named “files” and add a file named “name.txt” that contains your name or whatever you want.
Now make the build again ! and run it with this command :
docker run --rm -v ${PWD}/files:/app/files -e "NAME=steve" -p 8080:8080 hello_flask

Now open your browser and type localhost:8080/name_from_file , you should see the name that you already add it in the name.txt file. In this case, you can change the content of the file in real-time and reload the page, you should see the changes 😆.

Saving and publishing images

After finalizing your work there are two things you should do :

Saving images to tarballs or compressed archives
Publishing images to registries to be used in public or private…

Let's begin by saving our example image to a tarball by running this command docker save --output hello_flask.tar hello_flask , now, check your current directory and type in the terminal ls -sh hello_flask.tar to get the archive size. if you want to reduce the archive size use the gzip command which is compression tools. Execute this command :

docker save hello_flask | gzip > hello_flask.tar.gz

and check the size again 😃. you can now upload them to your cloud storage or wherever you want. Now if you want to load the tar file to the docker engine simply run :

docker image load -i=hello_flask.tar

Docker hub is our target now, to publish images in it you have to make an account first. the image name must be with this syntax account_name/image_name:image_tag so let's rename our image by running this command :

docker tag hello_flask hatembt/hello_flask:latest

Now you have to log in to your account and push the image to the public :

docker login
docker push hatembt/hello_flask:latest

To pull the image just run docker pull hatembt/hello_flask:latest , no need for credentials here 😃 .

Finally, I hope that this tutorial is helpful to everyone who wants to know docker.
Thank you 😃

Automate your Documentation with Gitlab and Mkdocs

hatem ben tayeb — Fri, 25 Dec 2020 11:17:11 +0000

Producing documentation may be painful and need a lot of time to write and operate. In this story, i will share with you, my way of generating docs using the devops approach. To make life easier, we will explore the art of automation 😃.

Let’s go folks 😙

Create a Gitlab repo

This is straightforward, follow these steps:

Log in to your GitLab
Click new project
Give it a name: auto_docs
Initialize it with a README.md file
Make it public or private
Hit create

Now clone the project by copying the URL and run this command :
$ git clone https://gitlab.com/auto_docs.git

Setting up the environment

I’m using a Linux environment but it is possible to reproduce the same steps on a Windows machine.
In order to follow me, you need a set of tools that must be available on your machine … make sure to to have python3 installed, I have python 3.8 (latest).

Creating a virtual environment

The easiest way to set up a virtual environment is to install virtualenv python package by executing pip install virtualenv .

Navigate to your local GitLab repository and create a new virtual environment.

$ cd auto_docs/ $ virtualenv autodocs $ source autodocs/bin/acivate

Installing Mkdocs Material

Make sure that the virtual environment is active.
Install the mkdocs material with this command : pip install mkdocs-material.
This package needs some dependencies to work .. install them by using a requirement.txt file, copy-paste the dependencies list to filename requirements.txt

Babel==2.8.0
click==7.1.1
future==0.18.2
gitdb==4.0.4
GitPython==3.1.1
htmlmin==0.1.12
Jinja2==2.11.2
joblib==0.14.1
jsmin==2.2.2
livereload==2.6.1
lunr==0.5.6
Markdown==3.2.1
MarkupSafe==1.1.1
mkdocs==1.1
mkdocs-awesome-pages-plugin==2.2.1
mkdocs-git-revision-date-localized-plugin==0.5.0
mkdocs-material==5.1.1
mkdocs-material-extensions==1.0b1
mkdocs-minify-plugin==0.3.0
nltk==3.5
Pygments==2.6.1
pymdown-extensions==7.0
pytz==2019.3
PyYAML==5.3.1
regex==2020.4.4
six==1.14.0
smmap==3.0.2
tornado==6.0.4
tqdm==4.45.0

Install them all with one command : pip install -r requirements.txt

Now it’s time to create a new mkdocs project 😅.

Run this command : mkdocs new . and verify that you have this structure :

|--auto_docs
    |--- docs
    |--- mkdocs.yml

The docs folder contains the structure of your documentation, it contains subfolders and markdown files.
The mkdocs.yml file defines the configuration of the generated site. Let's test the installation by running this command : mkdocs serve . The site will be accessible on http://locahost:8000 and you should see the initial look of the docs.

Setting up the CI/CD

let’s enable le CI/CD to automate the build and the deployment of the docs. Notice that GitLab offers a feature called gitlab pages that can serve for free a static resource (HTML, js, CSS). The repo path is converted to an URL to your docs.

Create the CI/CD file

Gitlab uses a YAML file — it holds the pipeline configuration.
The CI file content:

stages :
  - build
pages:
  stage: build
  image:
  name: squidfunk/mkdocs-material
  entrypoint:
    - ""
  script:
    - mkdocs build
    - mv site public
  artifacts:
    paths:
      - public
  only:
    - master
  tags:
    - gitlab-org-docker

This pipeline uses a docker executor with an image that contains mkdocs already installed… mkdocs build the project and put the build assets on a folder called site … to be able to use GitLab pages you have to name your job pages and put the site assets into a new folder called public.
For tags: check the runner's section under settings → CI/CD →Runners and pick one of the shared runners that have a tag GitLab-org-docker.
All things were done 🎉 🎉 😸 !

Oh ! just one thing … we forgot the virtual environment files .. they are big and not needed on the pipeline … they are for the local development only. The mkdocs image on the pipeline is already shipped with the necessary packages.
So … create a new file called .gitignore and add these lines:

auto_docs/ 
requirements.txt

The auto_docs folder has the same name as the virtual environment .. don't forget 😠! you will be punished by pushing +100Mi 😝 and you will wait your whole life to complete the process haha 😢.

Now run git add . && git commit -m "initial commit" a && git push… go to your GitLab repo and click CI/CD → pipelines, click on the blue icon and visualize the logs .. once the job succeeded, navigate to settings -> pages and click the link of your new documentation site (you have to wait for 10m~ to be accessible)

Finally, I hope this was helpful ! thanks for reading 😺 😍!

DEV Community: hatem ben tayeb

Looking for recommendations for provisioning a production AKS cluster

Scale your containers with ansible on a non kubernetes/swarm environment

Bash: writing a simple pod checker

Heroku: Deploy a dockerized Flask ML app

Naniiiiiiiiii !!

Hiding my nodejs application code within a docker container

Mailcow: Setting up a full featured self hosted mail server

For the reader

Mailcow

Server Requirements

Basic DNS configuration

Create an A record

Create the CNAME records

Create the an MX record

rDNS configuration

Security DNS Records

SPF record

DKIM record

DMARC record

Install mailcow

Configure Mailcow

Add your domain

Create a Mailbox for mymailserver.com

Add another domain to mailcow

Final thoughts

Delivering React .. The hard way !

**Daah ! let's start i can't wait 😍 !**

😜

Tools

😕Come on ! i was kidding 😄

Actually no ... contact me if you need some support

Architecture

😙

CI environment

**We are almost done 😍**

**. . .**

The react pipeline

**. . .**

Defining Secrets

**. . .**

**[KNOCK KNOCK](https://bongo.cat/) ... are you still here 😺**

**. . .**

Preparing Dockerfile

**. . .**

Deployment with ansible

**. . .**

Setting up the Pipeline

**. . .**

Final thoughts

**Thanks 😍**

Optimizing CI/CD Pipeline for Rust Projects (Gitlab & Docker)

What is Rust ?

What is DevOps ?

Identifying the problem

Static linking Vs Dynamic linking

Optimizing the CI/CD pipeline

The CI/CD pipeline

Deploying a Dockerized Angular App with Github Actions

What is devops ?

Plan

Code:

Build:

Test:

Release:

Deploy:

Operate:

Monitor:

Deploying an angular app

Preparing the deployment for heroku

Writing the Dockerfile

Preparing the deployment for the VPS on OVH

Preparing the pipeline

Hello Docker 😄

Common Docker commands

The Flask app

Environment variables in Docker

Volumes with Docker

Saving and publishing images

Automate your Documentation with Gitlab and Mkdocs

Daah ! let's start i can't wait 😍 !

Actually no ...
contact me if you need some support

We are almost done 😍

. . .

. . .

. . .

[KNOCK KNOCK](https://bongo.cat/) ... are you still here 😺

. . .

. . .

. . .

. . .

Thanks 😍