DEV Community: Ahmet Turkmen

from interview question to enlightenment: metaclasses in python

Ahmet Turkmen — Sun, 18 Jun 2023 10:20:00 +0000

metaclasses in python

https://mrturkmen.com/posts/metaclasses-python/

automate: run github ci/cd through slack slash command

Ahmet Turkmen — Sun, 15 Jan 2023 01:20:00 +0000

Easy integration for running Github workflow files through Slack slash command

automate: run github ci/cd through slack slash command | mrturkmen

Easy integration for running Github workflow files through Slack slash command

mrturkmen.com

packer: build custom images on cloud and local

Ahmet Turkmen — Sat, 17 Apr 2021 09:00:00 +0000

Build Custom Ubuntu 20.04 LTS on Local
- Anatomy of Packer Configuration File
- Builders
- Provisioner
- Post Processors
- Communicator
- How to run locally
Build Custom Ubuntu 20.04 LTS on Cloud
- Builders on Cloud
- Customize settins on Cloud
- How to run

THE REPOSITORY: https://github.com/mrtrkmnhub/ubuntu-packer

In this blog post, provisioning and customizing images using packer will be shown with a template repository.

If you are asking or wondering what is Packer, the official definition is :

Packer is a free and open source tool for creating golden images for multiple platforms from a single source configuration. (From Official Website).

This post includes provisioning of ubuntu image on AWS and local.

Build Custom Ubuntu 20.04 LTS on Local

In an ideal repository of Packer template, it would be nice to have a skeleton where it includes uploads, http, scripts folders along packer configuration file with a readme. Overall, the structure of folder might look like this :


    ├── http
    │ └── preseed.cfg # required to change defualt values of ubuntu image
    ├── readme.md # readme file to have instructions about what to do
    ├── scripts # scripts/ dir, includes scripts to run on custom image
    │ ├── cleanup.sh # cleans up /tmp 
    │ ├── install_tools.sh # installs custom tools
    │ └── setup.sh # setting up config in system wise
    ├── ubuntu-20.04.json # packer config for ubuntu 20.04
    └── uploads # directory to upload files to custom image 
        └── .gitkeep

In this setup, http/preseed.cfg defines answers to the questions which may be asked during installation of Ubuntu operating system. More information regarding to preseed.cfg file can be checked from its wiki

scripts folder composed of bash scripts, chef, ansible or any other installer configuration files or scripts which will install customized tools and define settings of ubuntu image.

uploads folder includes all files, deb packages, or any other files which will be copied to image which will be inside customized image.

Anatomy of Packer Configuration File

Any packer file composed of three main components which are ;

Builders

Define the desired platform and platform configurations, including API Key information and desired source images. Example snippet is given from the Packer file:


    "builders": [
    {
      "boot_command": [
        "<esc><wait>",
        "<esc><wait>",
        "<enter><wait>",
        "/install/vmlinuz<wait>",
        " auto<wait>",
        " console-setup/ask_detect=false<wait>",
        " console-setup/layoutcode=us<wait>",
        " console-setup/modelcode=pc105<wait>",
        " debconf/frontend=noninteractive<wait>",
        " debian-installer=en_US<wait>",
        " fb=false<wait>",
        " initrd=/install/initrd.gz<wait>",
        " kbd-chooser/method=us<wait>",
        " keyboard-configuration/layout=USA<wait>",
        " keyboard-configuration/variant=USA<wait>",
        " locale=en_US<wait>",
        " netcfg/get_domain=vm<wait>",
        " netcfg/get_hostname=ubuntu<wait>",
        " grub-installer/bootdev=/dev/sda<wait>",
        " noapic<wait>",
        " preseed/url=http://:/preseed.cfg<wait>",
        " -- <wait>",
        "<enter><wait>"
      ],
      "boot_wait": "10s",
      "format": "ova",
      "disk_size": 25240,
      "guest_additions_path": "VBoxGuestAdditions_.iso",
      "guest_os_type": "Ubuntu_64",
      "headless": true,
      "http_directory": "http",
      "iso_checksum": "sha256:f11bda2f2caed8f420802b59f382c25160b114ccc665dbac9c5046e7fceaced2",
      "iso_urls": [
        "iso/ubuntu-20.04.1-legacy-server-amd64.iso",
        "https://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/ubuntu-20.04.1-legacy-server-amd64.iso"
      ],
      "shutdown_command": "echo 'ubuntu'|sudo -S shutdown -P now",
      "ssh_password": "ubuntu",
      "ssh_port": 22,
      "ssh_timeout": "10000s",
      "ssh_username": "ubuntu",
      "type": "virtualbox-iso",
      "vboxmanage": [
        [
          "modifyvm",
          "",
          "--memory",
          "2048"
        ],
        [
          "modifyvm",
          "",
          "--cpus",
          "1"
        ]
      ],
      "virtualbox_version_file": ".vbox_version",
      "vm_name": "ubuntu_vm_ubuntu_20_"
    }
  ]

In the builders config, we are defining some set of keys in JSON file, which are very obvious from its name, we are considering to build image locally. All the keys are important in given builders config however most important and might need to update time to time is iso_urls which are the places where packer download iamges and customize it according to your scripts. Another crucial key is to have headless value true which means that there will be no GUI running when packer command is executed to run the Packer JSON file.

Provisioner

Defines how to configure the image most likely by your using existing configuration management tools like Ansible, Chef, Puppet or pure bash scripts.

In our example, bash scripts will be provided to install tools and update configuration of ubuntu image to make it customized. Provisioner section of a Packer JSON file can be seen as below:


 "provisioners": [
    {
      "type": "file",
      "source":"uploads",
      "destination": "/home/ubuntu"
    },
    {
      "execute_command": "echo 'ubuntu' | sudo -S -E bash ''",
      "script": "scripts/install_tools.sh",
      "type": "shell"
    },
    {
      "execute_command": "echo 'ubuntu' | sudo -S -E bash ''",
      "script": "scripts/setup.sh",
      "type": "shell"
    },
    {
      "execute_command": "echo 'ubuntu' | sudo -S -E bash ''",
      "script": "scripts/cleanup.sh",
      "type": "shell"
    }
  ]

Here we are defining existing bash scripts in order to execute in the process of customizing Ubuntu image. The steps under provisioners are pretty clear.

The content of uploads file will be uploaded to home directory /home/ubuntu

In second step, install_tools.sh will be executed and other steps will be followed in order.

Post Processors

Related to the builder, runs after the image is built, it is generally used to generate or apply artifacts. In this example, it is not required however more information can be found here: post processors

Communicator

How packer works on the machine image during the creation. By default it is over SSH communication and it does not need to be defined explicitly. More information can be found here: communicator

Over all packer file can be seen as follow:

{
    "builders": [
        {
        "boot_command": [
            "<esc><wait>",
            "<esc><wait>",
            "<enter><wait>",
            "/install/vmlinuz<wait>",
            " auto<wait>",
            " console-setup/ask_detect=false<wait>",
            " console-setup/layoutcode=us<wait>",
            " console-setup/modelcode=pc105<wait>",
            " debconf/frontend=noninteractive<wait>",
            " debian-installer=en_US<wait>",
            " fb=false<wait>",
            " initrd=/install/initrd.gz<wait>",
            " kbd-chooser/method=us<wait>",
            " keyboard-configuration/layout=USA<wait>",
            " keyboard-configuration/variant=USA<wait>",
            " locale=en_US<wait>",
            " netcfg/get_domain=vm<wait>",
            " netcfg/get_hostname=ubuntu<wait>",
            " grub-installer/bootdev=/dev/sda<wait>",
            " noapic<wait>",
            " preseed/url=http://:/preseed.cfg<wait>",
            " -- <wait>",
            "<enter><wait>"
        ],
        "boot_wait": "10s",
        "format": "ova",
        "disk_size": 25240,
        "guest_additions_path": "VBoxGuestAdditions_.iso",
        "guest_os_type": "Ubuntu_64",
        "headless": true,
        "http_directory": "http",
        "iso_checksum": "sha256:f11bda2f2caed8f420802b59f382c25160b114ccc665dbac9c5046e7fceaced2",
        "iso_urls": [
            "iso/ubuntu-20.04.1-legacy-server-amd64.iso",
            "https://cdimage.ubuntu.com/ubuntu-legacy-server/releases/20.04/release/ubuntu-20.04.1-legacy-server-amd64.iso"
        ],
        "shutdown_command": "echo 'ubuntu'|sudo -S shutdown -P now",
        "ssh_password": "ubuntu",
        "ssh_port": 22,
        "ssh_timeout": "10000s",
        "ssh_username": "ubuntu",
        "type": "virtualbox-iso",
        "vboxmanage": [
            [
            "modifyvm",
            "",
            "--memory",
            "2048"
            ],
            [
            "modifyvm",
            "",
            "--cpus",
            "1"
            ]
        ],
        "virtualbox_version_file": ".vbox_version",
        "vm_name": "ubuntu_vm_ubuntu_20_"
        }
    ],
    "provisioners": [
        {
        "type": "file",
        "source":"uploads",
        "destination": "/home/ubuntu"
        },
        {
        "execute_command": "echo 'ubuntu' | sudo -S -E bash ''",
        "script": "scripts/install_tools.sh",
        "type": "shell"
        },
        {
        "execute_command": "echo 'ubuntu' | sudo -S -E bash ''",
        "script": "scripts/setup.sh",
        "type": "shell"
        },
        {
        "execute_command": "echo 'ubuntu' | sudo -S -E bash ''",
        "script": "scripts/cleanup.sh",
        "type": "shell"
        }
    ],
    "variables": {
        "version": "0.1"
    }
}

How to run locally

This file can be run from the place where ubuntu-20.04.json file is located.

$ packer build ubuntu-20.04.json

It will start to build custom image by installing tools which are defined under scripts and configure username and password according to preseed.cfg and setup.sh files.

Build Custom Ubuntu 20.04 LTS on Cloud

It is more practical and preferrable to use if you already have an cloud option to consider. This packer configuration will create custom image directly on cloud and save it to AMIs to your AWS account.

The anatomy of packer files is similar, only section which needs to be changed compared to local one, is builders section. It is defining all required AWS variables and AMIs to customize.

As an cloud example AWS will be used to create custom image.

Builders on Cloud


"builders": [
        {
            "type":"amazon-ebs", 
            "region": "", 
            "access_key": "",
            "secret_key": "", 
            "subnet_id": "", 
            "security_group_id": "", 
            "source_ami_filter": {
                "filters": {
                    "virtualization-type": "hvm", 
                    "name": "ubuntu/images/*ubuntu-focal-20.04-amd64-server-*",
                    "root-device-type": "ebs"
                },
                "owners": ["099720109477"],
                "most_recent": true
            },
            "instance_type": "",
            "ssh_username":"ubuntu", 
            "ami_name": "ubuntu-ami-custom_"
        }

    ]

In this configuration, all keys are important to consider, however there are some which are crucial and required to run it. More information about the keys can be found here: Amazon AMI Builder

We would like to create a custom Ubuntu-20.04 image on cloud and save it as AMI to run it later, we are searching its pattern from available AMIs on AWS Management Console or it can be found through out this website : https://cloud-images.ubuntu.com/locator/ec2/

Once you have declared which AMI to customize, it needs to be located under source_ami_filter with wildcards and owners. Setting most_recent to true means that when this Packer JSON file is executed it will fetch and customize last updated AMI.

Access Key, Secret Key are required and should not be exposed to public in any moment, if exposed, they need to be updated immediately. They will be used to communicate with AWS to fire up instances to create custom image according to given settings defined in builders and provisioners.

The values of keys are defined in variables and parsed from out of it.


"variables": {
        "aws_access_key": "",
        "aws_secret_key": "",
        "aws_region": "",
        "aws_vpc": "",
        "aws_subnet": "",
        "ami_name": "",
        "ami_description": "",
        "builder_name": "",
        "username":"ubuntu",
        "instance_type":"t2.medium",
        "tarball": ""
    },

In variables section, username, instance_type, aws_access_key, aws_secret_key variables should be set correctly to create the image on cloud. Other variables are optional and variables section can be populated more.

Customize settins on Cloud

On cloud builds, cloud configuration file should be used instead of preseed.cfg to customize settings. The defaults.cfg file where it contains custom settings such as default username, password, changing visudo file and more. Example defaults.cfg can be as follow:

#cloud-config
system_info:
  default_user:
    name: ubuntu
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    lock_passwd: false
    plain_text_passwd: 'ubuntu'

More information regarding to defaults.cfg file can be found here and customized more: https://cloudinit.readthedocs.io/en/latest/topics/examples.html

How to run

Once variables are set, it can be run in same way with the local one.


$ packer build aws_packer.json

Complete packer JSON file : aws_packer.json

As a summary, Packer is really cool tool to use to automate the process of creating custom images and it can be used for Dockers as well. For local example in this post, it will produce OVA file to import, on cloud it will generate custom AMI under your AWS account.

All scripts and config files can be found in this repository: https://github.com/mrtrkmnhub/ubuntu-packer

fail2ban: block ssh bruteforce attacks 🇬🇧

Ahmet Turkmen — Wed, 24 Feb 2021 12:00:00 +0000

fail2ban

A while ago, I was checking servers’ logs to see any suspicious activities going on from outside. I noticed that the servers both staging/testing and production servers are receiving a lot of brute force SSH attacks from variety of countries which are shown in table below.

List of IP Addresses ( who are doing SSH Brute Forcing )

IP Address	Country Code	Location	Network	Postal Code	Approximate Coordinates*	Accuracy Radius (km)	ISP	Organization	Domain	Metro Code
171.239.254.84	VN	Ho Chi Minh City, Ho Chi Minh, Vietnam, Asia	171.239.254.0/23		10.8104,106.6444	1	Viettel Group	Viettel Group	viettel.vn
North Holland, Netherlands, Europe	159.65.192.0/20	1098	52.352, 4.9392	1000	Digital Ocean	Digital Ocean
117.217.35.114	IN	Bhopal,Madhya Pradesh, India, Asia	117.217.35.0/24	462030	23.2487,77.4066	50	BSNL	BSNL
Asia	113.164.79.0/24		9.7774, 105.4592	50	VNPT	VNPT				61.14.228.170
Da Nang, Vietnam, Asia	116.110.30.0/23		16.0685,
108.2215	1	Viettel Group	Viettel Group
43.239.80.181	IN	Kolkata, West Bengal, India, Asia	43.239.80.0/24	700006	22.5602, 88.3698	10	Meghbela Broadband	Meghbela Broadband	PMPL-Broadband.net
Tinh Thai Binh, Vietnam, Asia	14.255.136.0/23		20.4487,
106.3343	100	VNPT	VNPT	vnpt.vn
184.22.195.230	TH	Bangkok, Bangkok, Thailand, Asia	184.22.195.0/24	10310	13.7749, 100.5197	20	AIS Fibre	AIS Fibre	myaisfibre.com
116.110.109.90	VN	Da Nang, Da Nang, Vietnam, Asia	116.110.109.0/24		16.0685, 108.2215	20	Viettel Group	Viettel Group
Ho Chi Minh, Vietnam, Asia	115.76.168.0/23		10.8104,
106.6444	1	Viettel Group	Viettel Group	viettel.vn

** Information on the table gathered from: [https://www.maxmind.com/en/geoip-demo]

Ban failed attempts

Although servers have no password login, they are kept brute forcing on SSH port. Well, fail2ban was one of obvious solution to block those IP addresses permanently or temporarily. I prefered to block them all permanently until manual unblocking has been done by me.

The steps for installing fail2ban is pretty obvious, you are doing same things like, apt-get update && apt-get install fail2ban. After installation completed, configuration is much more important.

Following steps will guide you to block any ip address who are brute forcing on SSH.

Copy template file

   $ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Set Ban time

It is possible to set ban time permanent or temporarily. I preffered to setup permanent, so for this reason I have changed bantime = -1. Save and exit from the file when you are done.

$ vim /etc/fail2ban/jail.conf

# Permanent ban 
bantime = -1

Create custom rules for SSH


 $ vim /etc/fail2ban/jail.d/sshd.local

   [sshd]
   enabled = true
   port = ssh
   filter = sshd
   logpath = /var/log/auth.log # place of ssh logs 
   maxretry = 4 # maximum number of attempts that user can do

(*Maxretry value and log file can be changed according to your setup.)

Make the rules persistent

In order to make the rules persistent which means, the blocked IPs will not be deleted after restart of fail2ban service or restart of server. It requires to have some tricks to be done inside iptables rules under fail2ban. Add following cat and echo commands at the end of actionstart and actionban respectively .

$ vim /etc/fail2ban/action.d/iptables-multiport.conf 

                        .
                        .
                        .

actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> -m multiport --dports <port> -j fail2ban-<name>
          cat /etc/fail2ban/persistent.bans | awk '/^fail2ban-<name>/ {print $2}' \
          | while read IP; do iptables -I fail2ban-<name> 1 -s $IP -j <blocktype>; done

                       .
                       .
                       .

actionban = iptables -I fail2ban-<name> 1 -s <ip> -j <blocktype>
        echo "fail2ban-<name> <ip>" >> /etc/fail2ban/persistent.bans

Save and restart service

$ systemctl restart fail2ban

These are most basic steps to block IP addresses who are actively brute forcing to servers. After some time, I am able to see them with following command :)


$ sudo fail2ban-client status sshd

Status for the jail: sshd
|- Filter
| |- Currently failed:  12
| |- Total failed:  107
| `- File list: /var/log/auth.log
`- Actions
   |- Currently banned: 16
   |- Total banned: 16
   `- Banned IP list:   171.239.254.84 184.102.70.222 180.251.85.85 103.249.240.208 159.65.194.150 117.217.35.114 113.164.79.129 61.14.228.170 116.110.30.245 43.239.80.181 77.222.130.223 14.255.137.219 184.22.195.230 125.25.82.12 116.110.109.90 115.76.168.231

It is growing in time however at least they are not able to brute force the server with same IP addresses. There are plenty of other ways to make SSH port much more secure and effective however I think having updated ssh daemon/client, passwordless login and fail2ban will be enough in most of the cases. Therefore, while I was doing this stuff, although there are plenty of guides over there, I wanted to note down how I did it to come back and check if something happens.

Take care !

Deploy with Ansible on CI/CD 🇬🇧

Ahmet Turkmen — Sat, 12 Dec 2020 00:00:00 +0000

In this post, deployment process of an application with Ansible will be explained. Traditionally applications can be deployed in different ways, quite similar approach to deploy applications like in Ansible is executing bash script which has ssh commands. To give an example, Travis continuous integration has a feature where a bash script can be defined to deploy application and through given instructions within bash script, application can be successfully deployed.

Details regarding to deployment using Travis bash scripting can be found here

Travis Script Deployment

I would like to give an real case example from one of the project which I work on. We were using Travis script deployment for a while and it works pretty well. The bash script which I use in our deployment process is given below:

#!/usr/bin/env bash
f=dist/hknd_linux_amd64/hknd
amigo=./svcs/amigo
user=ntpd
hostname=sec02.lab.es.aau.dk
keyfile=./travis_deploy_key
deploy_path=/data/home/ntpd/daemon/hknd
amigo_path=/data/home/ntpd/daemon/svcs/amigo

if [-f $f]; then
    echo "Deploying '$f' to '$hostname'"
    chmod 600 $keyfile
    ssh -i $keyfile -o StrictHostKeyChecking=no $user@$hostname sudo /bin/systemctl stop hknd.service
    scp -i $keyfile -o StrictHostKeyChecking=no $f $user@$hostname:$deploy_path
    scp -i $keyfile -r -o StrictHostKeyChecking=no $amigo $user@$hostname:$amigo_path
    ssh -i $keyfile -o StrictHostKeyChecking=no $user@$hostname sudo /bin/systemctl start hknd.service
else
    echo "Error: $f does not exist"
    exit 1
fi

As you can observe from the bash script, every step of the deployment is given as ssh/scp commands. There is no harm regarding to it as long as it contains few steps. However, as time pass more configurations, applications will required to be deployed, updated, modified and checked, then it might turn into headache. Therefore, having well structured deployment steps using Ansible will put us to safe side.

Before jumping into deployment with Ansible, I would like to point out some factors which can be counted as disadvantages of not integrating Ansible to deployment process.

Not common way of utilizing resources
Not well structured deployment scripts which has high potential of being not working very well.
Having plain ssh commands increase likelihood of having issues regarding to settings, deployments and more.

There are many more drawbacks of using pure bash scripts in deployment process, however, these issues may not be applicable for all them.

For our case, I would like to convert our bash script given above to Ansible which has more elegant structure and easy to manage.

Move to Ansible

Since the bash script does not contain complex instructions, it would be very easy to convert it into Ansible playbooks. Before starting to convert it into Ansible playbook, necessary ssh connection should be set correctly for development and production environments. (- test environment as well if required -).

Setting ssh connection between server and ansible user is pretty straitforward, it contains following steps;

Generate SSH Key pair
Copy public key to authorized_keys on server side
Encrypt private key
Have decrypt script to use private key on CI without compromising it.

Overall simplified flow for deployment is given below :

As it is declared from overall picture above, we need to provide encrypted ssh key and script for decryption together, in order to use plain private key to access the server.

In this setup, Github CI will be control node which will have access to server where we would like to deploy the application.

Let’s start to complete steps,

Generate SSH Key pair

  $ ssh-keygen

You can keep everything default or provide some information about the questions when you run it. Once, execution of command finished, there will be public and private key, you need to append the public key to user’ authorized keys file on server. Afterwards, connection should be established, you may want to test it using traditional ssh command.

Encrypt Private Key

In order to use the ssh key which is generated before, we need to encrypt the key, I preferred to use gpg tool, there are many examples about it on internet, you can check it if you wish.

  $ gpg --symmetric --cipher-algo AES256 <private-key-file>

The command will prompt you to provide passphrase to encrpyt and decrypt the private key when required. Choose strong and long passphrase. Once it is done, include encrpyted file into git. (- which means commit it as well-)

Once they have completed, the rest is structing Ansible playbook to deploy the file to server.

Example Repo

I am going to create a repository on Github to demonstrate what I have described earlier in action.

For the demo purposes, I will upload a service file to server and start it, simplified version of given bash commands above.

Ansible playbook will contain following;

Stopping already running service
Changing binary file of the service
Starting it again

The tasks can be extended according to user needs however to keep it short and show how Ansible could be used on continuous integration, I will continue to have minimal playbook.

Link to example repository: https://github.com/mrturkmenhub/ansible-deploy

The structure of the repository as following:

As it can be observed from the figurre above, I have only three tasks which are combined under main.yml.

Some configuration regarding to Ansible, such as private key, inventory file location declaration is saved to file ansible.cfg among ssh connection configuration.

Inventory file contains server(s) to deploy the application.

This post is not about how to write ansible playbooks, hence, I am going to skip to explain it. If you would like to check and understand it you can check following repositories for examples;

Decrypt script is crucial file which is decrypting encrypted private key to access the server.

DO NOT FORGET TO SET YOUR SECRET_PASSPHRASE TO SECRETS OF THE REPOSITORY

The workflow file

The workflow file for this repository is pretty straitforward to create as well, what needs to be done is that ansible should be installed into environment. Afterwards, running ansible playbook command after decrpyting the encrypted private key will complete the tasks.

The generated workflow is for giving demonstration, in normal production case, the pipeline should NOT be broken, each step from testing to production deployment should be as much as automated.

The completed workflow file:

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the action will run. 
on:
  # Triggers the workflow on tagged commits  
  push:
    tags:
      - '*.*.*'

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

      - name: Install Ansible
        run: |
          sudo apt update -y
          sudo apt install software-properties-common -y
          sudo apt-add-repository --yes --update ppa:ansible/ansible
          sudo apt install ansible -y

      - name: Set Execute command to bash script
        run: chmod +x ./.github/scripts/decrypt.sh

      # Runs a single command using the runners shell
      - name: Decrypt large secret
        run: ./.github/scripts/decrypt.sh
        env:
          SECRET_PASSPHRASE: $

      - name: Escalate Private Key Permissions
        run: chmod 400 ~/.privkey

      - name: Run ansible command
        run: |
          ansible-playbook -i ./inventory main.yml
        env:
          ANSIBLE_CONFIG: ./ansible.cfg

      - name: Clean Key
        run: rm -rf ~/.privkey

The final result from Github actions:

Keep in mind that this is just a minor portion of a long pipeline which has all unit tests, checks, linting and integration tests. Without proper pipeline in place, having Ansible might not be logical or required. Consider your cases when you would like to move to deployment with Ansible.

Cheers !

Download Youtube Playlists and Release through Github Actions [ CI/CD ] 🇬🇧

Ahmet Turkmen — Tue, 24 Nov 2020 10:55:00 +0000

In some moments, Youtube algorithm is working perfect, but sometimes it shows a video from ten years ago from nowhere. For the moments where it shows and suggests videos/playlists to us, we might want to save the list of playlist and watch in some other time. It could be on a plane, train, bus, whenever you are planning to spent some time. However, taking the URL of a playlist and saving it to your cute note program might not be sufficient enough. There is a high chance that it will be forgotten or missed, therefore, I thought that it would be nice to have an automated way of saving playlists on somewhere and download them when I need. (- in particular, when there is no or limited internet connection -).

In this blog post, I will go through a simple project which downloads all the videos in a playlist and generates seperate tar.gz files for each playlist to release on Github using Github actions.

Requirements
Available Tools
Code Structure
Workflow File
Github Limitations
Repository
Demo

Requirements

Whenever starting a project, it is always nice to imply divide and conquer approach if you know what you would like to achieve. Divide and conquer approach will hugely assist you during the development and planning no matter what is the size of the project. As first step, let’s define what we need for accomblishing such a thing. ]

A library/program which downloads Youtube videos from given URL.
A library/program which compress the downloaded videos to minimize the size.
A workflow on Github Actions to trigger releases.

When the given components are clarified, only one more step left to have. There should be main component which combines the requirements given above. For this purpose, I will use Go programming language.

Available Tools

When existing libraries, tools and open source projects checked for the first requirement, there are some on Github, namely;

annie: 👾 Fast, simple and clean video downloader
youtube-dl: Command-line program to download videos from YouTube.com and other video sites
you-get: Dumb downloader that scrapes the web
ytdl: YouTube download library and CLI written in Go

These are the tools which enable users to download Youtube videos by providing the URL or the ID.(- some of them supports different social media platforms too, e.g vimeo -).

To make things simpler, I chose to use youtube-dl, since it is more promising than others and formats the output very well according to user output pattern.

Another point is to clarify which tool/library should I use to compress the downloaded videos from Youtube. With help of a little bit googling, I found out that pigz is quite nice tool which compress given folder/file in paralel by using all available cores of the machine. The second requirement is cleared as well, now it is time to combine both of them in one and add Github workflow on top it.

I will mention about the Github workflow file, after structure of the program which automates the process.

Code Structure

To make things faster (- in terms of development time -), I decided to go with using pre-existing binary file to execute commands, what I mean by that is basically having a pre-installed tool (youtube-dl & pigz) on the system before using this application.

If the readme file of youtube-dl is checked, youtube-dl can be installed as command line tool into your environment. It means that we can call the tool whenever we need from our application. There are other ways to accomblish it as well, such as instead of using pre-existing binary file, we can implement the functions in our application. However, the main idea of this post is NOT about how to create or use the library, the main idea is to present how it easy to have automated way of retrieving Youtube playlist videos and saving to Github Releases. The other requirement regarding to compress can be used in similar way. (- using a pre-existin command from system -)

To make things simple and extendable (- which means in case of more integration of tools we should be able to accomblish it without changing, many lines of code -), I will generate a main Client struct which will have exec function and it will be overridden according to command we pass.

The main client struct :

type Client struct {

    //youtube-dl client 
    YoutubeDL *YoutubeDL

     // Tar client 
    Tar *Tar

    // Used to enable root command
    sudo bool

    // flags to service
    flags []string

    // enable debug or not
    debug bool

    // Implementation of ExecFunc.
    execFunc ExecFunc

    // Implementation of PipeFunc.
    pipeFunc PipeFunc
}

Client struct has some fields which enables us to override whenever we want, the struct contains exec(cmd string, args ...string) ([]byte, error), shellPipe(stdin io.Reader, cmd string, args ...string) ([]byte, error), and shellExec(cmd string, args ...string) ([]byte, error) functions. It can be extended according to our requirements in the future. The explanations of the functions are given on top of functions inside the source code.

For the youtube-dl client, I implemented only a function (-the client functionalities are really easy to extend-), which downloads all videos on given playlist by using pre-existing command line tool youtube-dl.


package client

type YoutubeDL struct {
    c *Client
}

// exec executes an ExecFunc using 'youtube-dl'.
func (ytdl *YoutubeDL) exec(args ...string) ([]byte, error) {
    return ytdl.c.exec("youtube-dl", args...)
}

// DownloadWithOutputName generates Folder named with Playlist name
// downloads videos under given playlist url to Folder
func (ytdl *YoutubeDL) DownloadWithOutputName(folderName, url string) error {
    cmds := []string{"-o", folderName + "/%(playlist_index)s - %(title)s.%(ext)s", url}
    _, err := ytdl.exec(cmds...)
    return err
}

For any other additinal tool to use, it is extremely practical to add, for tar tool I have implemented following for specific purpose ( -which is compressing downloaded videos in paralel- ).


package client

type Tar struct {
    c *Client
}

// exec executes an ExecFunc using 'tar' command.
func (tr *Tar) exec(args ...string) ([]byte, error) {
    return tr.c.exec("tar", args...)
}

// CompressWithPIGZ using tar with pigz compress program to compress given data
func (tr *Tar) CompressWithPIGZ(fileName, folderToCompress string) error {
    cmds := []string{"--use-compress-program=pigz", "-cf", fileName, folderToCompress}
    _, err := tr.exec(cmds...)
    if err != nil {
        return err
    }
    return nil
}

Now, it is clear that for the both statements in the requirements section has been done. However, I wanted to keep track of what I have downloaded and release, for this reason, I have created two different csv files. They are called playlist-list.csv and old-playlist-list.csv under resources/ directory in the repository, playlist-list.csv will include all list of playlist URLs with preferred folder name to download. Futhermore, as you can guess, old-playlist-list.csv will include all the playlists which are downloaded and released. Once the playlist is downloaded and released with Github actions playlist-list.csv will be wiped and all content will be appended into old-playlist-list.csv file.

It will give easy way of checking what has been downloaded and released.

The code for reading and writing to csv files are pretty easy, and can be checked under main.go in the repository.

Workflow File

The workflow file will include some steps, which are;

Install pigz : required to compress data in parallel.
Install youtube-dl : required to download playlist from given URL.
Build Binary : required to have combined binary which handles both download and compress using pre-existing tools on the system.
Create Release : the step which initializes releases.
Run Binary : executes the program
Upload videos to Github releases : uploads downloaded content to releases.
Remove playlist and append downloaded playlists to old list : updates the list inside the playlist file and commits on master branch.

The steps given above are clickable to see inside the workflow on repository.

This small project is created for exclusive purpose, and it is very suitable to extend functionalities. However, there are many gaps regarding to the project such as;

it does NOT check the given playlists whether they have been already released or not.
it does NOT split created tar.gz files into 2 GB splits ( since it is required to have a file on Github releases under 2GB, but there is NO limitation for overall size of files on Github releases.)
Does NOT have error handling mechanism and more.

These are the points which appears when the project is checked at first glance, however there are more missing points which could be done. However, the main aim was to give idea how to accomblish automated way of downloading youtube videos and releasing with Github actions.

I am personally using it for personal needs whenever I find useful playlist, I include it into playlist-list.csv file and pushing the changes by tagging the commit in semantic versioning format.

There are tons of other services which could be integrated such as Slack, Discord, Mail or any another notification systems and more, however, to keep the post short and do not bother you, it is enough for now as it is.

The rule for the workflow could be easily changed, like instead of running it in tagged commits, it can run in scheduled way by changing run condition only, as shown below.

name: Download & Release Youtube Playlists

on:
  schedule:
    - cron: '0 0 * * *' # it means every day at midnight the workflow will run

If you require or would like to have more features, or fixes, suggestions and etc, you are more welcome to open issues.

Github Limitations

Since we are using Github actions, we have some limitations regarding to usage of it.

The limitations regarding to file sizes in releases, according to Github Statement here:Distributing large binaries

Basically, a file size which will be uploaded to releases should NOT exceeds 2 GB. However, keep in mind that it is per file, there is NO limitation for overall size of the release :). It means that repository will be updated to split files into chunks if size of the file exceeds 2 GB. So, in case of 15 GB of playlist, it should be uploaded in 2GB chunks to releases. (- a feature which is NOT exists on youtubeto yet -)

There are some more limitations:

Job execution time - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

Workflow run time - Each workflow run is limited to 72 hours. If a workflow run reaches this limit, the workflow run is cancelled.

More details about limmitations on Github Actions: Usage Limits

It is good to keep in mind the given limitations above.

Job execution time and Workflow run time can be easily fixed if you have your own server.

If you would like to run Github Actions in your server, there is no limitation regarding to Job execution time and Workflow run time.

Check out how to setup Github Actions for your server from here:

Setup self hosted runners

Repository

youtubeto: Automated Youtube PlayList Releaser

Demo

Latex with Github Actions🇬🇧

Ahmet Turkmen — Sun, 25 Oct 2020 16:25:00 +0000

In this post, I will be describing to setup a workflow to build and release your Latex files through Github actions. First of all, keep in mind that this post is not about what is Latex and how to use it.

It is extremely nice to integrate daily development tools such as CI/CD to your preparation of paper, without any hassle. Why is that because it is cool to track of what has been changed on a paper over time. In fact, having a couple of people who are responsible in different parts of paper, sometimes blocks others. Therefore, having such a workflow will increase productivity for everyone in a group. Whenever pull request created to main branch, it will be easy to check typos, logic errors and missing points by others.

Latex preparation
Setup Github Actions
Proof of Concept

Latex preparation

I am assuming that you have agreed to work on Latex template to complete a paper. In this case, there is only small step left to do, create a Github repository (-it should be on Github, Github Actions will be used-) and push all files of your Latex template. (-in general, in following structure-)

    |-sections 
        | introduction.tex
        | related_works.tex
        | problem.tex
        | solution.tex
        | conclusion.tex
    |- main.tex
    |- references.bib

The given example structure can be changed according to your wishes, however important and logical part is that having main.tex on root directory of repository.

Once it is set, there is only one step to complete which is setting up Github Action workflows.

Setup Github Actions

There are a few different Github Actions to use for compiling Latex document to PDF on marketplace. Most preferred one is https://github.com/xu-cheng/latex-action and it is quite easy to integrate and use.

It basically creates generated PDF file from provided Latex file, it can be set in workflow file as given below: (- Note that this workflow runs on tagged commits which has a tag with *.*.* pattern -)

name: Build LaTeX document
on: 
    tags:
        - '*.*.*' # semantic versioning 
jobs:
  build_latex:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Git repository
        uses: actions/checkout@v2
      - name: Compile LaTeX document
        uses: xu-cheng/latex-action@v2
        with:
          root_file: main.tex

However, setting up only this job is not sufficient enough to have completed workflow, we require to more jobs which are Create Release and Upload Release. As you may guess from their name, first one will create the release and second one will upload provided file to releases page. It can be setup as following


name: Release Compiled PDF 
on:
  push:
    tags:
      - '*.*.*'

jobs:
  build_latex:
    runs-on: ubuntu-latest
    steps:
      - name: Set up Git repository
        uses: actions/checkout@v2
      - name: Compile LaTeX document
        uses: xu-cheng/latex-action@v2
        with:
          root_file: main.tex

      - name: Create Release
        id: create_release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: $
        with:
          tag_name: $
          release_name: Release $
          draft: false
          prerelease: false

      - name: Upload Release Asset
        id: upload-release-asset 
        uses: actions/upload-release-asset@v1
        env:
          GITHUB_TOKEN: $
        with:
          upload_url: $ 
          asset_path: ./main.pdf
          asset_name: main.pdf
          asset_content_type: pdf

The given workflow is completed version of what you might have at the end. In summary, it builds PDF from provided Latex file, creates release and upload file to release. For more details, you can check information on each action page.

Proof of Concept

Here is example repository to check completed version.

https://github.com/mrturkmencom/latex-on-ci-cd

cool-kubernetes 🇬🇧

Ahmet Turkmen — Wed, 19 Aug 2020 22:37:00 +0000

Cool Kubernetes 🤓

It is a place which contains valuable resources about Kubernetes. It is somehow similar to awesome lists projects on Github, the logic is same however approach is different, instead of creating pure readme.md file like here, I am generating and publishing all stuff using Github Pages. Another important difference is that, this project does not only contain links to outsource, it also contains valuable information regarding to Kubernetes which is collected from variety of sources. For instance, let’s say someone is registered to Kubernetes or Kubernetes related course and realized that some figures, graphs or piece of information is valuable then that information could be added this resource hub. Many of you might say that “Well, you are duplicating existing information out there”, it is correct however, having most of valuable information from a place, which you know already, will save a lot of time for anyone who need some help.

For time being, the site does not contain lots of resources, however in time, it will contain troubleshooting techniques and approaches on Kubernetes.

The project is completely open source as any other awesome lists projects on Github, it means that the project is open to contribution as well, if you would like to add/improve/issue something on this resource hub. That’s perfect, do changes and create pull request or drop an issue 😉

I do not know whether it will be persistent or not, however I will try to update it occasionally.

It is completely open source project which is residing on cool-kubernetes website, any contribution is valuable.

Cheers !

Universal gRPC client demonstration [Evans] 🇬🇧

Ahmet Turkmen — Sat, 08 Aug 2020 09:19:00 +0000

In this post, I am going to write demo for a tool which I have just met, it is called Evans. It is basically universal gRPC client. What it means ? Basically when you have gRPC server and would like to test gRPC calls without creating client, you can test server side calls with Evans. It is known that gRPC is very common communication method between microservices, it can be used for internal and external communication. I do not have intention to explain what gRPC is in this post since it is not the purpose. If required documentation of gRPC can be investigated.

Create a simple gRPC server
- Sample repository
- Defining calls
- Compile Proto file
Run gRPC server
Demonstration of EVANS

Create a simple gRPC server

To demonstrate and see how evans works, a running gRPC should be exists, for this reason, I am going to provide a sample gRPC server. For this purpose, I will use Go programming language, however gRPC is supporting more programming languages which you may more familiar than Go.

gRPC server is basically an API endpoint where clients can make requests, since it is an API, first thing could be to define which methods will be used for this service.

For simplicity and purpose of this post, I have created a basic microservice which will has four calls namely, Add,Delete,List and Find. Since the purpose is to understand how evans works, the gRPC server does not need to be complex or includes lots of calls.

Sample repository

A sample microservice is created for demonstration purposes and all codes are available here : https://github.com/mrturkmencom/BookShelf

If you have already a running gRPC server, you can direcly pass to demonstration of evans, if not, you can clone https://github.com/mrturkmencom/BookShelf repository and test evans out.

Defining calls

In my opinion, it is always nice to prepare proto file before hand, because it is like a contract which creates your main service. Let’s imagine you would like to add, delete, list and find the books that you have read or wish to read. For this purpose, microservice should have at least four different calls which are Add, Delete, List and Find. There is no limit to have more calls however in order to do not get out of topic, I am keeping it small.

Following proto file would be enough for BookShelf service.


// it is important to declare syntax version
syntax = "proto3";
service BookShelf {
    rpc AddBook(AddBookRequest) returns (AddBookResponse) {}
    rpc ListBook (ListBooksRequest) returns (ListBooksResponse) {}
    rpc DelBook (DelBookRequest) returns (DelBookResponse){}
    rpc FindBook (FindBookRequest) returns (FindBookResponse){}
}

message AddBookRequest {
    BookInfo book = 1;
    message BookInfo {
        string isbn =1;
        string name =2;
        string author=3;
        string addedBy=4;
    }

}

message AddBookResponse {
    string message = 1;
}
message ListBooksRequest {
// no need to have anything
// could be extended to list books based on category ...
}

message ListBooksResponse {
    repeated BookInfo books =1;
    message BookInfo {
        string isbn =1;
        string name =2;
        string author=3;
        string addedBy=4;
    }
}

message DelBookRequest {
    string isbn =1;
}

message DelBookResponse {
    string message =1;
}
message FindBookRequest {
    string isbn =1;
}

message FindBookResponse {
    Book book = 1;
    message Book {
        string isbn =1;
        string name =2;
        string author=3;
        string addedBy=4;
    }
}

Once proto file is declared, it becomes more easy to continue. For the demostration purposes, I will store information of books in memory. However, as you know, it is NOT acceptable for any production level application.

Compile Proto file

proto files are great since once you have defined what you need, you can directly generate codes in available languages which are represented in gRPC supported languages. The generation of codes in your desired language is pretty straitforward, I am going to generate the code for Go programming language.

$ protoc -I proto/ proto/bs.proto --go_out=plugins=grpc:proto

It will generate ready to use Go source code for your rpc calls which are defined in proto file.

Afterwards, necessary piece of codes should be implemented, which are in memory store and book struct. For the aim of this post, I assumed that you have created all rest of the code as given in example gRPC server (-BookShelf-).

NOTE : Generating source code through protoc requires to have protoc tool to be installed before hand. Installation of protoc is over here

Run gRPC server

The post is not covering all aspects of gRPC, proto buffers, Go language and those are not the intention of this post. Therefore, I am assuming that you had gRPC server and would like to test out and see whether your proto contract is running correctly without creating client side codes. Once it is confirmed that your gRPC calls are running without encouraging any unseen problems, then creating client side code will be much easy without any problem.

You can start gRPC server with:

 $ go run server/main.go
 BookShelf gRPC server is running ....

Once gRPC server is up and running, you can use evans tool for inspecting gRPC server for available inquires.

Demonstration of EVANS

Evans is an open source project which is available at Github and I found it pretty useful, in particular, for people who have no idea what kind of calls are available in proto file, as it states its explanation, it is universal gRPC client. Installation of evans and more information is given its readme file.

It has plenty of features which are very handy to use for automating and testing some stuff on top of existing gRPC server or new one.

Let’s make a demo, when you are using evans your gRPC server should be up and running, in order to make communication with universal gRPC client - evans. I assumed that you have followed readme file of evans and installed it correctly.

Note that port 9000 is given because it is port of gRPC server.

❯ evans -r -p 9000

  ______
 | ____ |
 | | ____  ____ _ _ _____
 | __| \ \ / / / _. | | '_ \ /__ |
 | |____\ V / | (_| | | | | | \__ \
 | ______| \_/ \__ ,_| |_| |_| |___/

 more expressive universal gRPC client

BookShelf@127.0.0.1:9000> show services 

+-----------+----------+------------------+-------------------+
| SERVICE | RPC | REQUEST TYPE | RESPONSE TYPE |
+-----------+----------+------------------+-------------------+
| BookShelf | AddBook | AddBookRequest | AddBookResponse |
| BookShelf | ListBook | ListBooksRequest | ListBooksResponse |
| BookShelf | DelBook | DelBookRequest | DelBookResponse |
| BookShelf | FindBook | FindBookRequest | FindBookResponse |
+-----------+----------+------------------+-------------------+

BookShelf@127.0.0.1:9000>

As you can observe all calls which are used in proto file can be used through evans , moreover its usage is pretty straitforward.

You can check demonstration video below.

⚠️ You may wish to change the video quality to 1080p60

https://youtu.be/GnAUkPUXYCs

If you have any questions, fix, or something else, do not hesitate to contact with me.

NGINX Ingress Controller with HAProxy for k8s cluster 🇬🇧

Ahmet Turkmen — Fri, 10 Jul 2020 16:35:00 +0000

In recent post, which is Setup Highly Available Kubernetes Cluster with HAProxy 🇬🇧, a highly available Kubernetes cluster is created. However, once I started to dig in and deploy some stuff to cluster, I realized that I am not able to connect any deployed application or services. For instance, when an web application is deployed using HAProxy load balancer (endpoint), and check from kubectl (on client side), its status is running. However, that application could not be reached from outside world although I re-patch an external IP address by following command

 $ kubectl patch svc <application-name> -n <name-of-namespace> -p '{"spec": {"type": "LoadBalancer", "externalIPs":["<haproxy-ip-address>"]}}'

After some searching and reading, I realized that worker nodes require their own ingress controllers in order to forward traffic between them in case of load. I will be giving more information of how I fix the issue, however let’s learn some basic terms and general information about ingress controller.

What is ingress controller ?
Updates to cluster
- Setup NGINX Ingress Controller
- Steps to create NGINX Ingress controller
- Deploy Example Application

What is ingress controller ?

The best and simple explanation to this question is coming from Kubernetes official documentation over here, as they are expressing that ;

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontend to help handle the traffic.

Whenever you have services which are running inside a cluster and would like to access them, you need to setup ingress controller for that cluster. The missing part was having no ingress controller on worker nodes in my k8s cluster. Everything was working however there was no access to them from outside world, that’s why ingress controller should take place in cluster architecture.

In this post, I will go for NGINX ingress controller with its default setup, however there are plenty of different ingress controllers which you may go for. I might change NGINX to Traefik in future but it depends on requirements yet for now, I will go with nginx ingress controller. The reason is that, it is super easy to setup, super rich with different features, included Kubernetes official documentation and fulfill what I am expecting for now.

Updates to cluster

Let’s briefly what I have explained in previous post;

Create VMs
Setup SSH connection
Use KubeSpray to deploy cluster
Create HAProxy and establish SSH connection with all nodes.

I have noticed that when deploying cluster, some add-ons should be enabled in order to use ingress controller from cluster with external HAProxy load balancer. Now, since cluster deployment was established with Ansible playbooks, it is not needed to setup everything from scratch. All modified configuration can be re-deployed without effecting any resource which is exists on cluster setup. It means that, I can enable required parts in configuration file and re-deploy cluster as I did on previous post.

Enable ingress controller from inventory file inside KubeSpray

 $ vim inventory/mycluster/group_vars/k8s-cluster/addons.yml
 # Nginx ingress controller deployment
 ingress_nginx_enabled: false -> true

Once this configuration part is updated from existing KubeSpray configuration files, k8s cluster should be redeployed with same command in previous post

Assumption : previous configured KubeSpray settings are used.

 $ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

It will take a while and update all necessary parts which are required.

Include Ingress API object to route traffic from external HAProxy server to internal services

To include Ingress API object, HAProxy configuration file should be modified, following lines should be added to /etc/haproxy/haproxy.cfg file.

  $ vim /etc/haproxy/haproxy.cfg

      frontend kubernetes-ingress-http
          bind *:80
          default_backend kubernetes-worker-nodes-http

      backend kubernetes-worker-nodes-http
          balance leastconn
          option tcp-check
          server worker1 10.0.128.81:80 check fall 3 rise 2
          server worker2 10.0.128.137:80 check fall 3 rise 2
          server worker3 10.0.128.156:80 check fall 3 rise 2

In given configuration balancing algorithm is leastconn which can be changed into any load balancer algorithm which is supported by HAProxy, however leastconn algorithm is fitting more to what I would like to achieve that’s why it is declared as leastconn. Note that this configuration addition is on top of added part on previous post.

Once HAProxy configuration is updated, HAProxy should be restarted systemctl restart haproxy. It is all for HAProxy configuration, now let’s dive into setting up NGINX Ingress Controller.

Setup NGINX Ingress Controller

It is super simple to deploy and setting up NGINX ingress controller since it is well documented and explains required parts in detail. To setup NGINX Ingress Controller, I will follow official guideline which is exists on NGINX Ingress Controller Installation.

Image is taken from (https://www.nginx.com/products/nginx/kubernetes-ingress-controller/#resources)

In normal cases, the situation is as given figure above, however, since in existing k8s cluster, I am using HAProxy for communicating with clients, I need NGINX ingress controller inside worker nodes which will manage running applications/services by communicating with HAProxy and eventually, the services will be accessible from outside world.

If I summarize how overview diagram will look like in my case is like in given figure below.

It can be observed that, in given k8s cluster overview, HAProxy is in front, it communicates with clients, afterwards transmitting request based on defined rule on HAProxy configuration. Each worker node has NGINX ingress controller, what exactly it means, whenever a request appear to cluster, worker nodes will agree between each other and response back to user without having any problem. Since NGINX ingress controller is capable of load balancing inside worker nodes as well.

There is also Ingress Resource Rules part inside cluster, what it does, is that all routing rules based on path forwarded given service, an example on this is given below.

Steps to create NGINX Ingress controller

All steps shown below for installation of NGINX Ingress Controller taken from https://docs.nginx.com/nginx-ingress-controller/installation/

Make sure that you are a client with administrator privilege, all steps related to NGINX ingress controller should be done through kubectl (on client computer/server)

Clone Ingress Controller Repo

$ git clone https://github.com/nginxinc/kubernetes-ingress/
$ cd kubernetes-ingress

Create a namespace and a service account for the Ingress controller

$ kubectl apply -f common/ns-and-sa.yaml

Create a cluster role and cluster role binding for the service account

$ kubectl apply -f rbac/rbac.yaml

Create a secret with a TLS certificate and a key for the default server in NGINX

$ kubectl apply -f common/default-server-secret.yaml

Create a config map for customizing NGINX configuration:

$ kubectl apply -f common/nginx-config.yaml

Afterwards, there are two different ways to run NGINX ingress controller deployment, which are as daemonset or as deployment. Main difference between those are summarized on official installation page as;

Use a Deployment. When you run the Ingress Controller by using a Deployment, by default, Kubernetes will create one Ingress controller pod.

Use a DaemonSet: When you run the Ingress Controller by using a DaemonSet, Kubernetes will create an Ingress controller pod on every node of the cluster.

I will go with DaemonSet approach, the reason is that generally when you have background-ish tasks which will run non-stateless then DaemonSet is more preferred way of running it.

$ kubectl apply -f daemon-set/nginx-ingress.yaml

Once it is applied as daemon set, the result could be checked with following command and result will be similar to given result below.

$ kubectl get all 
NAME READY STATUS RESTARTS AGE
pod/nginx-ingress-47z8r 1/1 Running 0 24h
pod/nginx-ingress-cmkfq 1/1 Running 0 24h
pod/nginx-ingress-ft5pv 1/1 Running 0 24h
pod/nginx-ingress-q554l 1/1 Running 0 24h
pod/nginx-ingress-ssdrj 1/1 Running 0 24h
pod/nginx-ingress-t9jml 1/1 Running 0 24h

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/nginx-ingress 6 6 6 6 6 <none> 24h

Deploy Example Application

To test how an application will be exposed to externally from k8s cluster, an example applicaton could be deployed as given below. Note that the following example is simplest example for this context, hence, keep in mind that it might require more configuration and detailed approach then described here when you would like to deploy more complex applications.

Create a sample NGINX Web Server (Using provided example)

nginx-deploy-main.yml

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Taken from https://kubernetes.io/docs/tasks/run-application/run-stateless-application-deployment/

In given yaml deployment file above, two replicas of NGINX:1.14.2 will be deployed to cluster and it has name of nginx-deployment. The yaml explains itself very well.

It can be deployed either through directly from official link or from your local depends on your preferences.

$ kubectl apply -f https://k8s.io/examples/application/deployment.yaml

## or you can do same thing with local file as given below
$ kubectl apply -f nginx-deploy-main.yml

Expose deployment:

$ kubectl expose deploy nginx-deployment --port 80

Once it is deployed to cluster and exposed, there is one step left for this simple counter example is that, exposing the service and creating ingress rule (resource) in yaml file, by specifiying kind as Ingress.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  rules:
  - host: <dns-record> (a domain like test.mydomain.com)
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deployment
          servicePort: 80

The crucial part is serviceName and servicePort which are defining specifications of the services within cluster. The yaml specifications can be expanded as shown below, assume that you have wildcard record in your domain name server and have multiple services which are running in same port in a cluster, yaml file can be re-defined as given below.

nginx-ingress-resource.yml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-controller
spec:
  rules:
  - host: <dns-record> (a domain like test.mydomain.com)
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deployment
          servicePort: 80
      - path: /apache
        backend: 
           serviceName: apache-deployment
           servicePort: 80
      - path: /native-web-server 
        backend:
           serviceName: native-web-server-deployment
           servicePort: 80

Keep in mind that all given services should be deployed before hand otherwise when a request made to any path which is not deployed, it may return either 404 or 500. There are plenty of different options to define and update the components in a k8s cluster. Therefore, all yaml files should be changed according to requirements.

Create ingress controller rules from provided yaml file

$ kubectl create -f nginx-ingress-resource.yml

Now, the NGINX web server deployment is ready on given DNS record in yaml file and according to request paths different services can be called which are also running inside kubernetes cluster.

Note that, provided yaml files are just simple example of deploying NGINX web server without any certification, when certificates (HTTPS) enabled or any other type of deployment happened different configurations should be applied.

When everything goes without any problem, you will have a cluster which uses NGINX Ingress controller for internal cluster routing and HAProxy as communication endpoint for clients. Keep in mind that whenever a new service or deployment take place, required configuration should be enabled in HAProxy configuration as it is enabled for port 80 applications above. Different services will have different requirements therefore it is important to catch main logic in a setup. It is all done for this post.

Cheers !

Setup Highly Available Kubernetes Cluster with HAProxy 🇬🇧

Ahmet Turkmen — Sun, 05 Jul 2020 17:43:00 +0000

The main purpose of this blog post a simple walkthrough of setting up Kubernetes cluster with external HAProxy which will be the endpoint where our kubectl client communicates over. Node specifications for this setup is given as shown in the table below. Keep in mind that all of them has access to each other with password and without password. The environment which Kubernetes cluster will stay is running on OpenStack. It means that once a configuration (ssh keys, hosts, and etc) is done for example master 1 then all other nodes could be initialized through snapshot of master 1. To be able to setup such a Kubernetes cluster easily, I will be using KubeSpray which is a repository where it has all required configuration and playbooks for setting up necessary cluster.

Node Specification
Prerequisites
General Overview
KubeSpray Configuration
External Load Balancer Setup (HAProxy)
Setup KubeSpray Configuration

The intention of this walkthrough is that setting up your own Kubernetes cluster in your own servers, this post is not very useful for people who are already using cloud provider solutions.(Kubernetes cluster as a service). You can checkout following resources listed below : (few of them :) )

Cloud Providers Solutions:

Node Specification

Kubernetes cluster will be setup on following nodes in the table below, note that HAProxy will run on another node and all ansible playbooks and setting up Kubernetes cluster will be managed through HAProxy. Keep in mind that all nodes + HAProxy is under same subnet internally which means that we will only one external IP address where HAProxy use and kubectl clients communicate. All instances are running on ubuntu_18.04, it means that the instructions and steps may not work with another system.

Prerequisites

Nodes
Requirements of KubeSpray
Setting up SSH Key Across Nodes
Getting snapshot ( -it is optional -)
Setting up login with password

General Overview

The following sketch is general overview of how Kubernetes cluster will look like at the end of this walkthrough, the figure is super overviewed version of cluster.

In given figure above, nodes do not have any external IP adress however, including HAProxy, all of them in same subnet, only HAProxy has external IP address which will be reachable by kubectl clients.

Before moving installation step of Kubernetes cluster, we need to setup a sample master node (instance) with predefined configuration. Since we will have only one server which is open to outside world, we need to make sure that there is a connection between HAProxy and sample master node. I am currently calling it sample master node, it is because, preliminary configurations such as authentication with password, disabled swap area and ssh keys will be all configured. This sample master node should be started and accesible over HAProxy, which means that in order to access to sample master node, I should do following;

SSH to HAProxy using SSH key ( Password Login disabled ) like ssh -i ~/.ssh/id_rsa <username>@<ha-proxy-external-ip>
Copy SSH Key to HAProxy, which let you in to sample master node
Then SSH to sample master node with same approach. (ssh ~/.ssh/masternode.pem <username>@<master-node-ip>

After you are inside sample master node, now, some configurations and setting should be done. Afterwards, we can initialize other five nodes from snapshot of configured sample master node.

Steps:

Enable Password Login if not enabled already.

$ echo "PermitRootLogin yes" >> /etc/ssh/sshd_config
$ sed -i -E 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config

Specify Password for ROOT

$ sudo su 
$ passwd

Given commands will ask new unix password for root user. Define the password and do not forget or lose it. Since we will gonna use snapshot of this configured machine, all settings will be same, I did like that to shortcut the process.

Disable swap area (RUN ALL COMMANDS AS ROOT)

$ swapoff -a

Afterwards, exit from sample master node, create snapshot of that node (it is called volume snapshot in OpenStack), once you have successfully created snaphot, all five other nodes should be initialized from snapshot of this sample master node. This way, there is no need to repeat same steps described above.

In case of not having possibility to create snapshot follow given steps (if and if only, you could NOT create snapshot and initialize other five nodes from the snapshot)

Create all nodes (workers and masters)

Enable SSH connection to all nodes from HAPRoxy server.

From HAProxy server, execute following steps. (-Make sure that you have configured SSH connection with ROOT priviledges and have access to all nodes from HAProxy node -)

Once you are sure that you have SSH access to all nodes from HAProxy through SSH, implement following steps.

Install parallel-ssh (-to run a command in parallel on nodes-) (run with ROOT priviledges)

$ apt-get update && apt-get install -y pssh

Install HAProxy (as ROOT priviledges)

$ apt-get install -y haproxy

Modify /etc/hosts (-For easy communication through nodes-)

Append worker and master node IPs to /etc/hosts file

$ vim /etc/hosts 
10.0.128.156 worker3
10.0.128.137 worker2
10.0.128.81 worker1
10.0.128.184 master3
10.0.128.171 master2
10.0.128.149 master1

Create nodes text file on home directory

$ cat nodes 
worker3
worker2
worker1
master3
master2
master1

Since IP addresses of them defined in /etc/hosts file, system can now recognize and connect IPs of them through just by name

Generate and Copy SSH Key to all nodes (Required for easy communication)

If there is already a SSH key (like in ~/.ssh/id_rsa), you can use it as well.If not, you can do following step

$ ssh-keygen # will prompt passphrase, you can leave empty , NOTE THAT IF YOU DO NOT HAVE SSH KEY, GENERATE IT.

$ for i in $(cat nodes); ssh-copy-id $i; done

The for loop given as second command will copy ssh key to all nodes, then accesing any node without password will be flawless.Like given command below;

$ ssh master1 # in defualt uses same username with terminal session

Disable swap area on all nodes (Note that if you are using snapshot method, no need to do this step)

$ parallel-ssh -h nodes -i "swapoff -a"

Parallel SSH tool is handy to complete tasks in parallel for multiple hosts.

KubeSpray Configuration

KubeSpray is a repository to setup Kubernetes clusters with predefined configuration settings using Ansible playbooks. The usage of KubeSpray is pretty straightforward, as default settings, KubeSpray is using internal load balancers in each worker node, which means that when you setup a Kubernetes cluster using default values of KubeSpray, you will have following arch overview.

However, in this guide, external load balancer approach will be used to setup cluster, if you wish to leave everything as default with KubeSpray, you can skip this External Load Balancer Setup part.

External Load Balancer Setup (HAProxy)

Modify configuration file of HAProxy to enable external LoadBalancer, copy this following configuration and append to /etc/haproxy/haproxy.cfg. (end of file)

listen kubernetes-apiserver-https
  bind <your-haproxy-internal-ip>:8383
  mode tcp
  option log-health-checks
  timeout client 3h
  timeout server 3h
  server master1 <your-master1-ip>:6443 check check-ssl verify none inter 10000
  server master2 <your-master2-ip>:6443 check check-ssl verify none inter 10000
  server master3 <your-master3-ip>:6443 check check-ssl verify none inter 10000
  balance roundrobin

Balance algorithm is roundrobin however you can change it from list of available balance algorithms provided by HAProxy.

Once it is done, save and restart HAProxy service.

$ systemctl restart haproxy

Setup KubeSpray Configuration

Since external load balancer will be used, there is few things to be done to change default values in KubeSpray. Following steps will be done on HAProxy node.

Clone the project and prepare environment


$ git clone https://github.com/kubernetes-sigs/kubespray
$ apt-get install -y python3-pip # install pip3 if not installed
$ cd kubespray

Follow the guide on KubeSpray README.md file

Following instructions taken from KubeSpray README.md

# Install dependencies from ``requirements.txt``
sudo pip3 install -r requirements.txt
# Copy ``inventory/sample`` as ``inventory/mycluster``
cp -rfp inventory/sample inventory/mycluster
# Update Ansible inventory file with inventory builder
declare -a IPS=(10.0.128.149 10.0.128.171 10.0.128.184 10.0.128.81 10.0.128.137 10.0.128.156)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

Modify generate hosts YAML file

When you check inventory/mycluster/hosts.yaml file, you will notice that it created two master nodes, which we require three, add missing one properly to that list as shown below.

all:
  hosts:
    master1:
      ansible_host: 10.0.128.149
      ip: 10.0.128.149
      access_ip: 10.0.128.149
    master2:
      ansible_host: 10.0.128.171
      ip: 10.0.128.171
      access_ip: 10.0.128.171
    master3:
      ansible_host: 10.0.128.184
      ip: 10.0.128.184
      access_ip: 10.0.128.184
    worker1:
      ansible_host: 10.0.128.81
      ip: 10.0.128.81
      access_ip: 10.0.128.81
    worker2:
      ansible_host: 10.0.128.137
      ip: 10.0.128.137
      access_ip: 10.0.128.137
    worker3:
      ansible_host: 10.0.128.156
      ip: 10.0.128.156
      access_ip: 10.0.128.156
  children:
    kube-master:
      hosts:
        master1:
        master2:
        master3:
    kube-node:
      hosts:
        master1:
        master2:
        master3:
        worker1:
        worker2:
        worker3:
    etcd:
      hosts:
        master1:
        master2:
        master3:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

Once it is done, the other thing which should be modified to use external load balancer HAProxy, is all.yaml file located under inventory/mycluster/group_vars/all/.

all.yml is general configuration file which specifies main configurations of your cluster, it uses Nginx load balancer by default which means that each worker node has its own local nginx load balancer as given second figure above. If not specified anything else.

Disable default load balancer

$ vim inventory/mycluster/group_vars/all/all.yml
loadbalancer_apiserver_localhost: false

Add external load balancer HAProxy.

$ vim inventory/mycluster/group_vars/all/all.yml
## External LB example config
apiserver_loadbalancer_domain_name: "<domain-name-of-lb>"
loadbalancer_apiserver:
  address: 10.0.128.193
  port: 8383

Initialize cluster deployment

# under kubespray/ directoy 
$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

It will take around 10-15 minutes which depens on your cluster and if everything goes well, at the end of deployment through Ansible you will not face with any problem. If so, you can test it by SSH to master node and try kubectl cluster-info.

$ kubectl cluster-info
Kubernetes master is running at .....

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

It means that Kubernetes cluster with three master and three worker nodes available to use.

Note that the default configuration of cluster could be changed more however before attempting to change default configuration, make sure that you did correct research on what to change on KubeSpray default settings. Otherwise, there might be problems regarding to customized configuration settings.

For more information stay updated and watch KubeSpray regarding to issues, pitfalls and more.

Last step for this post is creating kubectl configuration for your personal/work computer to access the cluster. Install kubectl on your environment. Afterwards copy configuration from master node to your ~/.kube/ as config.

Since we have only one endpoint, configuration file should be copied to HAProxy Server then your computer, through rsync or scp

On HAProxy Server

$ scp root@master1:/etc/kubernetes/admin.conf config # will copy admin.conf as config 
$ cp config /home/ubuntu/ # copy to a user home dir
$ chown ubuntu:ubuntu /home/ubuntu/config # change owner of the file

On your personal/work computer

$ scp -i ~/.ssh/haproxy.pem ubuntu@<ha-proxy-ip>:/home/ubuntu/config ~/.kube/

Now, you should be able to get and dump your cluster information as in master nodes.

$ kubectl cluster-info
Kubernetes master is running at .....

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

There are lots of configurations and different settings regarding to Kubernetes cluster environment and generally using Cloud Provider solutions are less painful or painless. However, sometimes it is less costly to setup your own environment and having full access to anything could be better for learning under the hood things or creating highly customized environments. It really depends on your situation therefore it is up to you to go and setup your own Kubernetes cluster or use it as service from cloud providers.

By the way, thanks for giving time to checkout the post 😉

Kendimize özel VPN kurulumu 🇹🇷

Ahmet Turkmen — Wed, 01 Jul 2020 10:00:00 +0000

VPN Kuralım
- Neden kendi VPN sistemi kurmalıyız ?
- Sunucu Ayarları
- Kullanıcı Ayarları
- Güvenlik Duvarı ayarları

VPN Kuralım

Bugün sizlere kendinize ait VPN sistemi nasıl kurulur, onu anlatmak istiyorum, daha önce İngilizce olarak, yayınladım fakat Türkçe bir kaynağın da faydalı olabileceğini düşündüm. Burada anlatılanlar, ubuntu ailesine (16.04,18.04) ait sunucular üzerinde test edilmiştir.

İlk olarak bulut hizmeti sağlayan bir şirketten bu DigitalOcean, Google Cloud, Microsoft Azure veya Amazon olabilir, sunucu kiralıyorsunuz, en ucuzu ve makul olanı DigitalOcean tarafından sunulan aylık 5 dolar olan sunucu diyebilirim. Sunucuyu kiraladıktan ve ssh bağlantısını sağladıktan sonra VPN kurulumuna geçebiliriz.

VPN hakkında tam bilgisi olmayan arkadaşlar için şu şekilde özetlenebilir, sizin için oluşturulmuş sanal bir bağlantı noktası gibi düşünebilirsiniz. Yani VPN’e bağlandıktan sonra bilgisayarınızdan çıkan ve bilgisayarınıza gelen ağ trafik şifrelenmiş olarak işlenir. Üçüncü parti yazılımların veya MITM gibi saldırıların önüne geçmiş olursunuz.

Neden kendi VPN sistemi kurmalıyız ?

Çünkü şu anda var olan bütün VPN sistemleri, ücretsiz olarak hizmet sağlasa dahi, sizin bilgilerinizin satılması, arşivlenmesi ve gerektiğinde ilgili birimlere aktarılması amacıyla kaydedilmektedir. Bunun ne gibi zararları olabilir gelin birlikte şöyle bir sıralayalım:

Oltalama saldırılarına sadece sizin bilebileceğiniz bilgiler ile maruz kalma.
Ziyaret ettiğiniz siteler tarafından reklam bombardımanına maruz kalma.
Kişisel bilgilerinizin reklam veren ajanslara satılması, bu durum birçok kişi tarafından tam olarak anlaşılamıyor, yani şu şekilde anlaşılamıyor, internet üzerinden alışveriş yapan A kişisi, kendine ait bilgilerin, onun bilgilerini satacak kişiler tarafından değersiz olduğuna inanıyor ve hiçbir gizlilik sağlamadan internet kullanımına devam ediyor. Bu sonunda o kişiye zarar vermese bile o kişinin konuştuğu, görüştüğü veya birlikte çalıştığı arkadaşlara zarar verebiliyor.

Burada sıralananlar sadece buzdağının görünen ucu bile diyemeyiz, günümüzde veri işleme teknikleri ve yaklaşımları öyle gelişmiştir ki siz bile kendinize ait olan bir şeyin varlığına farkında olmadan onlar işlemleri tamamlamış oluyor :).

Bu ve bunlardan çok daha fazla nedenden dolayı VPN kullanımı şart diyebilirim. Peki bunu nasıl yapacağız, bu kısımdan sonra sizin bir bulut sağlayıcısı tarafından sunucunu kiraladığınızı ve ssh bağlantısını sağladığınızı varsayıyorum.

Bu gönderide WireGuard VPN uygulaması kullanılacaktır. WireGuard VPN uygulaması açık kaynaklı bir uygulama olup, sağladığı imkanlar sayesinde diğer VPN uygulamalarına (OpenVPN ve diğerleri) kıyasla çok daha hızlı ve güvenilirdir.

Sunucu Ayarları

VPN uygulamasını kiraladığımız sunucu üzerine kuralım.

$ sudo apt-get update && sudo apt-get upgrade -y 
$ sudo add-apt-repository ppa:wireguard/wireguard
$ sudo apt-get update 
$ sudo apt-get install wireguard

Uygulamayı çekirdek güncellemeleri ile birlikte güncellemek için gerekli komutu girelim.

  $ sudo modprobe wireguard

Aşağıda verilen komut girildiğinde beklenen sonuç.

$ lsmod | grep wireguard

wireguard 217088 0
ip6_udp_tunnel 16384 1 wireguard
udp_tunnel 16384 1 wireguard

Anahtarları üretelim

$ cd /etc/wireguard
$ umask 077
$ wg genkey | sudo tee privatekey | wg pubkey | sudo tee publickey

VPN Konfigurasyon dosyasını /etc/wireguard/wg0.conf ayarlayalım.


[Interface]
PrivateKey = <daha-öncesinde-üretilen-gizli-anahtar>
Address = 10.120.120.2/24
Address = fd86:ea04:1111::1/64
SaveConfig = true
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o ens3 -j MASQUERADE
ListenPort = 51820

Burada önemli nokta ens3, ip tables komutu içerisinde yer alan en3, sunucudan sunucuya farklılık gösterebilir, bundan dolayı sizin sunucunuzda ne ise ağ kartının ismi onu girmelisiniz. ifconfig komutu sayesinde öğrenilebilir.

Bir diğer önemli nokta ise daha öncesinde 4. adımda üretilen privatekey, içeriğinin PrivateKey alanına girilmesidir.

Ağ trafiğini yönlendirme

/etc/sysctl.conf dosyası içerisine aşağıda verilen bilgileri girerek kaydediniz.

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Bilgiler gerekli dosyaya kaydedildikten sonra aşağıdaki komutlar sırası ile girilmelidir.

$ sysctl -p
$ wg-quick up wg0

Komutların girilmesi ve herhangi bir sorun görülmemesi durumda ve wg komutu terminale girildikten sonra aşağıda verilen çıktıya benzer bir çıktı göreceksiniz.

$ wg
interface: wg0
  public key: loZviZQpT5Sy4gFKEbk6Vc/rcJ3bH84L7TUj4qMB918=
  private key: (hidden)
  listening port: 51820

Eğer herhangi bir sorun ile karşılaşmazsanız bu adıma kadar, bu demek oluyor ki, sunucu tarafında işiniz şimdilik tamamlandı geriye sadece kendi bilgisayarımızı, telefonumuzu vs VPN sunucusuna bağlamak kaldı.

Kullanıcı Ayarları

Kullanıcıların kendi bilgisayar ortamlarında, telefonlarında, tabletlerinde veya diğer sunucularında kullanabileceği uygulamaları buradan indirebilirsiniz.

Gerekli uygulamayı kendi ortamınıza indirdikten sonra tek yapmanız gereken, VPN sunucusu tarafında ayarladığımız VPN’e bağlanmak, bunun için gerekli olan sadece konfigurasyonları doğru girmek olacaktır.

Kullanıcı tarafında, uygulama üzerinden aşağıda verilen konfigurasyona benzer bir ayarı (kendi kurduğunuz VPN ayarlarına göre privatekey ve ip adressi değişiklik gösterecektir.) ayarlamanız gerekmektedir.

[Interface]
Address = 10.120.120.2/32
Address = fd86:ea04:1111::2/128
# note that privatekey value is just a place holder 
PrivateKey = KIaLGPDJo6C1g891+swzfy4LkwQofR2q82pFR6BW9VM=
DNS = 1.1.1.1

[Peer]
PublicKey = <sunucunuza-ait-public-anahtar>
Endpoint = <sunucunuzun-dış-ip-adresi>:51820
AllowedIPs = 0.0.0.0/0, ::/0

Gerekli işlemler kullanıcı tarafında da sağlandıktan sonra, sunucu tarafında bu kullanıcıya bağlantı izni vermek kalıyor, onuda aşağıda verilen komut ile sağlayabilirsiniz.

$ wg set wg0 peer <kullanici-public-anahtari> allowed-ips 10.120.120.2/32,fd86:ea04:1111::2/128

Sunucu tarafindan kullanicinin VPN baglantisi sağladığını aşağıda verilen komut ile teyit edebilirsiniz.

$ wg

interface: wg0
  public key: loZviZQpT5Sy4gFKEbk6Vc/rcJ3bH84L7TUj4qMB918=
  private key: (hidden)
  listening port: 51820

peer: Ta9esbl7yvQJA/rMt5NqS25I/oeuTKbFHJu7oV5dbA4=
  allowed ips: 10.120.120.2/32, fd86:ea04:1111::2/128

Daha sonrasinda, wireguard tarafından oluşturulan ağ kartını aktivate edelim.

$ wg-quick up wg0

Güvenlik Duvarı ayarları

Bazen sunucu tarafında yapmanız gereken bazı güvenlik duvarı ayarları bulunmakta, bunlar VPN bağlantısını başarılı bir şekilde sağlamanız için kritik öneme sahiptir.

$ ufw enable

VPN uygulamasına bağlanmamızı sağlayacak portu açıyoruz.


$ ufw allow 51820/udp

IP tabloları ile 51820 portu için bazı ayarlamalar yapıyoruz.

$ iptables -A INPUT -p udp -m udp --dport 51820 -j ACCEPT
$ iptables -A OUTPUT -p udp -m udp --sport 51820 -j ACCEPT

Burada önemli olan kısımlardan biriside bütün komutlar ROOT, yani yönetici yetkisi ile yapılmalı, aksi takdirde hata verecektir.

Bu noktadan sonra, bilgisayarınıza, tabletinize veya telefonunuza kurduğunuz WireGuard uygulaması sayesinde sorunsuz ve güvenlikli bir şekilde internetinizi kullanabilirsiniz.