DEV Community: Sunish Surendran K

Control your playbook Task execution

Sunish Surendran K — Tue, 12 Sep 2023 08:43:57 +0000

In Ansible, fork, serial, throttle, and async are configuration parameters that control the concurrency and parallelism of task execution

Fork

It specifies the maximum number of parallel connections that Ansible will use when executing tasks. Increasing the fork value allows you to parallelize task execution across multiple hosts.
For example, setting fork: 3 means Ansible will execute tasks on up to 3 hosts simultaneously as shown below.

Serial

The serial parameter in an Ansible playbook allows you to control the number of hosts that are targeted at a time during playbook execution. It's useful for scenarios where you want to limit the rate of change or avoid overloading hosts.

For instance, if you set serial: 3, Ansible will target and execute tasks on three hosts at a time, waiting for them to complete before moving on to the next set of hosts.

Throttle

The throttle parameter in an Ansible playbook allows you to control the rate at which tasks are executed. In below example for Task 2 only one machine is executed at a time.

Async

The async parameter in an Ansible task allows you to run that task asynchronously. It means Ansible will start the task against specific set of machines and immediately move on to the next set of machines without waiting for first set to complete.

You can also use the poll parameter in conjunction with async to specify how often Ansible should check the status of the asynchronous task. In the below example poll is provided as two, so every two second the ansible will check the status of the task executed in the machines

I end this blog with a question - If you want to achieve a scenario where a machine jumps to Task 2 as soon as its Task 1 is completed, while other machines are still executing Task 1.How you structure your playbook to achieve this scenario? Please comment if you know the answer.

Creating Python Packages

Sunish Surendran K — Sun, 25 Jun 2023 14:16:08 +0000

TLDR

Prior to delving straight into the creation of Python packages, let's first grasp the concept of Modular programming.

Modular programming refers to the process of breaking a large, unwieldy programming task into separate, smaller, more manageable subtasks or modules. Individual modules can then be cobbled together like building blocks to create a larger application.

So, what are the example for modular programing in python?
Functions, modules and packages are all constructs in Python that promote code modularization.

NICE!!

You may already know what is a function in python, but did you get confused thinking what is the difference between module and package in python?

Suppose you have developed a very large application that includes many modules. As the number of modules grows, it becomes difficult to keep track of them all if they are dumped into one location. This is particularly so if they have similar names or functionality. You might wish for a means of grouping and organizing them.

In this case Packages allow for a hierarchical structuring of the module namespace using dot notation.

OK, that great! Let's think we have some modules available; Now let's see how we can create a package out of it!

Python packages are created using python packaging tools like setuptools, Flit, PDM, Poetry etc..

Python setuptools is a widely used tool for packaging Python projects. Today our plan is to create package using setuptools.

Step 1 : Setup your environment

Upgrade pip tool

python -m pip install --upgrade pip

Upgrade Build

python -m pip install --upgrade build

This installs following packages in your machine build-0.10.0, packaging-23.1, pyproject_hooks-1.0.0, tomli-2.0.1

Build package is used to build the python package

Install setuptools

pip install setuptools

you can read about setuptools in the link https://pypi.org/project/setuptools/

Install twine tool

pip install twine

Step 2 : Create folder structure as shown below

Here I created a master folder call sunish_package under it another folder level src/sunish_package and test

Also you can I created files called LICENSE, ReadME and pyproject.toml

Step 3: Add modules

Under the folder src/sunish_package i created two files with name init.py and mdoule_1.py

init.py is the Package Initialization script, it can be empty
module_1.py is our module with test code

def test_function():
    print("Hello World")

Step 4: Update pyproject.toml file

In older version of setuptool the user need to create setup.py script in the root location of the package which contains all the configuration for the package, but starting from setuptools version 58.0.0, it introduced support for reading package metadata and build configurations from the pyproject.toml file.

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "sunish_package"
version = "0.0.1"
authors = [
  { name="Example Author", email="author@example.com" },
]
description = "A small example package"
readme = "README.md"
requires-python = ">=3.7"
classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]

[project.urls]
"Homepage" = "https://github.com/pypa/sampleproject"
"Bug Tracker" = "https://github.com/pypa/sampleproject/issues"

The important thing to note here is the name of the package, it should be pointing to the same folder name under src. Also you can see under [build-system] title I have filled requires section as setuptools. This indicate the python package need to be build using the setuptools.

Step 5: Update LICESNSE and ReadME.md

ReadME.md

# About Package

This is an example package

LICESNSE

Copyright (c) 2023 Sunish.

I prefer you to use correct License and update ReadME with your package information. Since I am creating a test package, here I am filling it with basic information.

Step 6 : Create Python Package

Go to your root folder and run the below command.

python -m build .

This created two folders called dist and sunish_package.egg-info

Whl file is the file we want, the python packages are packaged in the forrmat called wheels (.whl)

Step 7 : Import and test

Go the dist folder and install the package using pip

pip install .\sunish_package-0.0.1-py3-none-any.whl

you can check the package is listing in the pipe list with below command

pip list -v

Now open a python console and import our new package

from sunish_package import module_1
module_1.test_function()

SUCCESS!! it prints "hello world" as expected.

Step 8 : Push the package to internal artifactory python repository

In order to push the package to our internal artifactory we make use of the tool Twine.

Create the .pypirc file in the dist folder and update the artifactory url , username and password

[distutils]

index-servers = local

[local]

repository: http://localhost:8081/artifactory/api/pypi/pypi

username: <user_name>

password: <password>

Then run the below command to push the package to the artifactory repo

twine upload -r local <PATH_TO_THE_FILES> --config-file ~/.pypirc

Behind the Automation of Jurassic Park

Sunish Surendran K — Fri, 19 May 2023 21:19:17 +0000

The iconic Jurassic Park movie, released in 1993, captivated audiences with its mesmerizing depiction of a fully automated park powered by computer programming. While we marvelled at the breathtaking scenes, have you ever wondered which programming language was utilised to bring the park's automation to life?"

While it is not explicitly stated in the Jurassic Park movie that the control system language used in the park is Pascal, there are indeed references and similarities to Pascal in the film.

In the movie, the character Dennis Nedry, portrayed as a skilled computer programmer, is shown writing code for the park's control system. The code snippets shown on screen display Pascal-like syntax and structure, suggesting a resemblance to the Pascal programming language.

Additionally, the character Ray Arnold, played by Samuel L. Jackson, mentions "Pascal routines" when discussing the park's control system. This further suggests that Pascal may have been an inspiration for the fictional language depicted in the movie.

However, it's important to note that the language portrayed in the film as "Nedry's Code" or "JP-CL" is a fictional representation and does not correspond directly to the real Pascal programming language. The movie takes artistic liberties and simplifies the technical aspects for storytelling purposes.

A dive into Ansible Modules and Module Utilities

Sunish Surendran K — Tue, 09 May 2023 15:45:17 +0000

I have been tasked with developing a custom script to be executed on a target machine via Ansible. While aware of Ansible's capability to integrate custom scripts, I am uncertain whether such integrations are accomplished via Ansible Modules or Plugins ?

After quick research in google I understood that Module is designed to handle a specific task and provides a standardized interface for Ansible to communicate with the remote systems.

Plugins, on the other hand, are used to extend the functionality of Ansible itself, rather than defining the desired state of a system. Plugins can be used to add new functionality to Ansible, such as custom inventory scripts, custom connection types, and custom callback plugins. Plugins are generally written in Python and can be loaded by Ansible at runtime.

So now I know I need to create module not plugin, now the question is in which language I need to write the module Python or PowerShell? It's depending upon the machine you are targeting. If you are planning to run the module against a Linux machine, then python is the best choice. In my case I am targeting a windows machine, so I need write down a windows module.

Suddenly a question popped up on my mind why I can't write a python module and use it with windows and Linux, that is the best way right?

It is not possible. Ansible team explained the reason in below link.

https://docs.ansible.com/ansible/latest/os_guide/windows_faq.html#can-i-run-python-modules-on-windows-hosts

ok, then let's create ansible PowerShell module.

When I read through the documentation I understood we should use ansible inbuild **Module_Utils **while creating custom ansible modules.

Why to use ansible Module_Utils?

Module_utils can be used by Ansible module developers to simplify their code and reduce duplication. By using module_utils functions, module developers can avoid writing boilerplate code and instead focus on implementing the core functionality of their module.

For example, the ansible.module_utils.basic module provides functions for handling input and output, while the ansible.module_utils.urls module provides functions for making HTTP requests.

Where I can find this ansible Module_Utils?
You can find it under the path where ansible is installed.

How to import Module_Utils to my custom module?

Like I shown below you can import Module_Utils for python and PowerShell module



#!/usr/bin/python
from ansible.module_utils.basic import AnsibleModule



#!powershell
#AnsibleRequires -CSharpUtil Ansible.Basic



#python
module = AnsibleModule(argument_spec=spec)



#powershell
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec)

Note: spec parameter is a hashtable/dictionary that defines the argument specification for the module.

In both Python and PowerShell ansible module you can see that I created an object called module for the class AnsibleModule.

The AnsibleModule class is a core component of Ansible that provides a standard interface for writing Ansible modules.

AnsibleModule class can be used to access the input parameters and the output results of the module. The object module created provides several methods and properties, including:

Params: A hashtable that contains the input parameters passed to the module.
ExitJson($result): A method that exits the module and returns the output result in JSON format.
FailJson($msg): A method that exits the module with an error message in JSON format.
Warn($msg): A method that writes a warning message to the console.
Fail($msg): A method that writes an error message to the console and exits the module.

Using all these methods I created a nice PowerShell module for ansible.
I don't want to explain how I written the module because it is already mentioned in the ansible documentation https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general_windows.html

So now start writing your own ansible modules. :) enjoy!!

Memory management in Python

Sunish Surendran K — Thu, 09 Mar 2023 20:16:05 +0000

Python's memory management is efficient and transparent, and developers do not need to worry about managing memory manually.

In python, the Primitive variable objects like integers, floating-point numbers, boolean values are typically stored in the stack. Also Function calls and local variables, function call stack and exception stack are stored in the stack.

Storing these objects on the stack allows for efficient memory management. It is worth noting, however, that not all primitive values are stored on the stack. For example, large arrays or other data structures that exceed a certain size may be stored on the heap instead. Additionally, Python's memory management is dynamic and flexible, so the actual storage location of an object can depend on a variety of factors, including the object's size, lifetime, and usage patterns.

Objects, lists, dictionaries and other data structures are stored on the heap. Now you may get confused thinking the primitive data types also is an object, but it is stored in heap. Then what type of objects stored in Heap?

Objects created from a class are typically stored on the heap, not the stack. This is because class instances are typically long-lived objects that can exist beyond the scope of a single function call.

When you create an instance of a class using the class constructor (i.e. by calling the class with arguments), a new object is created on the heap. The class instance contains all the data members defined in the class and any additional data members that are added dynamically at runtime. The memory required for the object is dynamically allocated and managed by the Python runtime.

For example, consider the following code:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

person1 = Person("John", 30)

In this example, person1 is an object of the Person class that is created on the heap when the Person constructor is called.

Let's see how we can create a Stack Overflow and Heap Overflow in Python

Here is an example of a recursive function that could cause a stack overflow:

def recursive_function(x):
    if x == 0:
        return 0
    else:
        return x + recursive_function(x-1)
result = recursive_function(10000)
print(result)

In this example, the recursive_function calls itself recursively with smaller values of x until it reaches the base case where x == 0. However, if x is too large, the function will create a large number of nested function calls on the call stack, eventually causing a stack overflow.

Allocate a large amount of memory using a list

large_list = [0] * 1000000000

In this example, we create a list with a billion elements by using the multiplication operator to repeat the value 0 a billion times. Since each element in the list takes up memory, this can cause a heap overflow if the system does not have enough memory to allocate a billion elements.

Python also includes a garbage collector that automatically frees up memory that is no longer being used by the program. This helps to prevent memory leaks and makes it easier to write Python programs that don't have to worry about managing memory manually.

Python's garbage collector runs automatically in the background, and most of the time, you don't need to worry about it. However, in some cases, it can be helpful to manually trigger the garbage collector using the gc.collect() function.

Stack and Heap memory

Sunish Surendran K — Tue, 07 Mar 2023 19:33:16 +0000

Before Jumping to know about Stack and Heap memory let's do a time travel to past.

The Past

The concept of heap and stack memory is an important part of computer programming and memory management. The origins of these concepts can be traced back to the early days of computer science and programming languages.

One of the earliest references to the concept of a "stack" in computer programming is attributed to John von Neumann, who developed the concept of a "last-in, first-out" (LIFO) data structure in the 1940s.

John von Neumann

The idea of using a stack to manage memory in computer programs was later popularized by the ALGOL programming language in the 1950s.

The concept of heap memory can also be traced back to the early days of computer science. The term "heap" was first used in reference to a data structure in the 1960s, and was popularized by the programming language Lisp. The idea of dynamic memory allocation, which allows programs to allocate memory at runtime, was also introduced around this time.

Overall, the concepts of heap and stack memory have evolved over time as computer hardware and programming languages have advanced. Today, they remain an essential part of memory management in modern programming languages and operating systems.

Also, there are some programming languages that are designed to work without heap and stack memory. For example, the "Forth" programming language, which was first developed in the 1970s, uses a unique memory management system that does not rely on heap or stack memory. Instead, Forth programs use a single, contiguous block of memory to store both program code and data.

Now let's come back & check what is this Stack and Heap

When RAM is manufactured, it is initially unallocated, which means that it does not have any data stored on it.

However, the operating system of a computer, such as Windows or Linux, is responsible for managing the allocation of memory.

When a program is executed, the operating system allocates memory to the program, which includes both stack and heap memory. The stack memory is typically allocated at the top of the program's address space, while the heap is allocated at the bottom.

The size of the stack is usually predetermined at compile time, and the operating system sets aside a fixed amount of memory for the stack. When a function is called, the processor allocates a block of memory on the stack for the function's stack frame, which includes the function's parameters, local variables, and return address. As the function executes, it pushes and pops data on and off the stack.

If we try to store dynamically generated data on the stack, it can cause a stack overflow because the stack has a limited size. The amount of memory required for dynamically generated data may be unpredictable and may exceed the available stack space, leading to unpredictable behavior and program crashes.

Therefore, dynamic data is usually stored in the heap, which is a larger pool of memory that can be allocated and deallocated at runtime. The heap allows us to allocate memory for dynamically generated data as needed, and to release that memory when it is no longer needed, without worrying about stack limitations.

I hope this gave you an idea about stack and heap memory. If you think there is something more, please make a comment.

Hey wait..One more interesting thing to share, the cover page for this blog is generated by DALL-E " A deep learning models developed by OpenAI to generate digital images from natural language descriptions"

Quality Metrics

Sunish Surendran K — Sun, 05 Mar 2023 08:16:48 +0000

Code Coverage

The percentage of code that is covered by (automated) unit tests.

Abstract Interpretation

Let's explain the abstract with an example.

if (obj == null) {
    obj = getObject();
}
obj.method();

Here in the above example there is one problem abstract interpretation may detect that is the obj variable may have the value null even after line 2.

Cyclomatic Complexity

Programs with a lot of conditional statement or loops are more complex. This means maintainability of the program is hurt, since it is hard to understand for new developers working on it.

Compiler Warnings

Compiler warnings are potential errors in the source code that are discovered during compilation. These include syntactic errors for which the semantics are easily misinterpreted, portability issues and type errors.

Coding Standards

Coding standards provide rules for software developers to follow while writing code. The most important reason for coding standards is maintainability; a clear set of rules makes it easier for programmers to understand the meaning of code.

Code Duplication

Code Duplication is a software metric that indicates the amount of source code that occurs more than once in a program.

Fan Out

The fan out metric indicates how many different modules are used by a certain module.

For Python, the number of modules mentioned in the import statement are counted. from-import counts as 1.

Thoughts on :Nginx

Sunish Surendran K — Sun, 14 Nov 2021 15:41:36 +0000

Fantastic way to wrangle some good software product is to experience it firsthand.So lets try Nginx today.And i have a weired hobby of connecting my fav software products with comic characters.

I have choosen Green Latern as Nginx
"All the powers that the Power Ring gives"

Nginx [engine x] is a free and opensource high performance web-server that can be used as

Reverse proxy
Load balancer
SMTP proxy
TCP/UDP proxy server.

There is a commercial edition available for Nginx called as Nginx Plus.

Source Code:

Nginx source code is published on mercurial http://hg.nginx.org/nginx/file/tip , May be its new info for your, Mercurial is a free multiplatform distributed version control system. Mercurial is used via the command line; the program file itself is called hg.So like git we can download the nginx code using below command line.

hg clone http://hg.nginx.org/nginx

If you want to build the nginx from the source code please follow the link http://nginx.org/en/docs/howto_build_on_win32.html

NGINX Configuration file

NGINX can be configured with nginx.conf file, By default this file is named as nginx.conf and placed under /usr/local/nginx/conf, /etc/nginx, or /usr/local/etc/nginx.

The Nginx configuration file consist of Directives and Contexts

What are Directives and Contexts?

NGINX configuration consists of key-value pairs called directives. Directives decides which configuration to apply.They can be organized and grouped into Blocks known as Contexts.

Contexts are tree like structures that can be nested within one another and Directives can only been used within Contexts.

Directives and Context looks like below:

Nginx Architecture

Nginx has one master process and several worker processes.

The main purpose of the master process is to read and evaluate configuration, and maintain worker processes.

Worker processes do actual processing of requests. nginx employs event-based model and OS-dependent mechanisms to efficiently distribute requests among worker processes.

The number of worker processes is defined in the configuration file and may be fixed for a given configuration.

IF we mention auto like below the number of worker processes automatically adjusted to the number of available CPU cores

Each worker process can open a by default 512 connections

As previously mentioned, nginx doesn't spawn a process or thread for every connection. Instead, worker processes accept new requests from a shared "listen" socket and execute a highly efficient run-loop inside each worker to process thousands of connections per worker

Extending Nginx:

Nginx can be extended by using thrid-party modules.There is a large ecosystem of third‑party modules, ranging from language interpreters to security solutions.

Have a look at the below nginx blog about compling dynamic modules for nginx
https://www.nginx.com/blog/compiling-dynamic-modules-nginx-plus/

Understanding GitHub Copilot(AI Pair Programmer)

Sunish Surendran K — Thu, 01 Jul 2021 12:03:12 +0000

GitHub Copilot is an AI pair programmer which suggests line completions and entire function bodies as you type.

"Hmm that's Cool right!! But i think there is lot of kerfuffle around it."

GitHub Copilot is powered by the OpenAI Codex AI system.

"You might me now thinking what is this OPenAI Codex AI system."

OpenAI is an AI research and deployment company.The investors include Microsoft, Reid Hoffman’s charitable foundation, and Khosla Ventures.Goal of the company is promoting and developing friendly AI in a way that benefits humanity as a whole.

Recently OpenAI transitioned from non-profit to for-profit.
They have developed so many products.They have papers submitted on topic generative pre-training of a language model.This Generative pre-training commonly known by its abbreviated form GPT.

So now coming back to our GitHub Copilot, it is built on a new algorithm called OpenAI Codex, which OpenAI CTO Greg Brockman describes as a descendant of GPT-3.While GPT-3 generates English, OpenAI Codex generates code.

It works best with Python, JavaScript, TypeScript, Ruby, and Go, according to a blog post from GitHub.

Since now GitHub is with Microsoft , Microsoft can easly push the OpenAI elements to GitHub.For now, Microsoft is only offering the service that knows about code stored in public repositories.Future we can expect that for enterprise customers too.

GitHub have a nice documentation written for the Copilot.Have a look.

"Now the question will be how to use it"

It is available today as a Visual Studio Code extension.

But currently we need to signup and access is limited to a small group of testers during the technical preview.

WSL,VSCode Terminal: "Coddled and Cosseted"

Sunish Surendran K — Sun, 09 May 2021 12:33:20 +0000

FLAG: "If you have more time as a developer to improve your Terminal look and also if you're a Windows user, this blog can help you!"

Here we try out some fancy stuff for the WSL terminal.

The plan is to use WSL via windows Terminal. For this first install WSL and Windows Terminal on your machine.

Go through the below 6 steps and have the awesome terminal on your machine.

STEP 1 : Install zsh in WSL

apt-get install zsh

STEP 2 : Install "Oh My Zsh"

visit the website https://ohmyz.sh/#install

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

STEP 3 : Install PowerLevel10k

git clone https://github.com/romkatv/powerlevel10k.git $ZSH_CUSTOM/themes/powerlevel10k

STEP 4 : Install Font FiraCode NF (We need this font then only the PowerLevel10k theme looks fine.)

Download the Font from the link

After downloading the ttf file into your machine, just double-click the ttf file. The file will open, then press the install button at the top. This will install the font on your windows machine.

Now we need to map the font to the WSL Terminal, for that go to the settings of the terminal.

Click on the Open JSON file

The JOSN file looks like below one

Copy the below code and paste it in your JSON file.

NOTE: update the guid with the guid of your ubuntu instance

            {
                "fontFace": "FiraCode NF",
                "guid": "{YourUbuntuguid}",
                "hidden": false,
                "name": "Ubuntu",
                "snapOnInput": true,
                "source": "Windows.Terminal.Wsl",
                "useAcrylic": true
            },

You need to copy the code to the appropriate section of the JSON file as shown below.

STEP 5 : Configuring the Powerlevel10k

Go to WSL and open the .zshrc file kept in your home directory.

vi ~/.zshrc

Add the below line and save the file

ZSH_THEME="powerlevel10k/powerlevel10k"

Reload the .zshrc file in your terminal

source ~/.zshrc

Then the configuration screen for the powerlevel10k will appear if not type the below command

p10k configure

Follow the instructions and set up the new look as below!!!

ohhhh!!! Wait!! We have a few more work. Now we need our VScode terminal also looks cool.

STEP 6 : Updating the VSCode terminal font

Open VScode and go to its setting

Search the below line

Terminal integrated font family

update the font name here also.

Done, if you open the WSL terminal in vscode it looks cool!!.
Enjoy!

Deploying Kubernetes Operators using Operator Life Cycle Manager

Sunish Surendran K — Sat, 20 Mar 2021 09:11:43 +0000

TL;DR

Welcome to the world of Kubernetes Operators!!

With the introduction of Custom Resource Definitions (CRDs) in version 1.7 Kubernetes is more extensible and developers are bringing more light toward the Operator world.Nowadays most of the applications are deployed as operators and they will have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.

So in order to make the development and maintenance of operator easier RedHat introduced an Operator framework, it has SDK, olm(operator lifecycle manager), and operator registry.

In this article, let's explore how we can deploy an operator to the Kubernetes cluster with the help of the Operator lifecycle manager.

Creating Kubernetes Cluster using KIND

"kind is a tool for running local Kubernetes clusters using Docker container nodes"

KIND is spawned from a Kubernetes Special Interest Group and I can say confidently, it’s going to be a very useful tool.

OK, for now just download the binaries of KIND and then run the below command in your machine. This will bring up a new Kubernetes cluster with name testcluster.

Since this article is not about KIND I am not going to explain all pre-requisites to be installed to run KIND, you can refer to the KIND documentation.

Kind create cluster –name testcluster –config=kindconfig.yaml

Here I used a kindconfig.yaml because I wanted to map few extra ports to my base machine.(Optional)

Within minutes Kubernetes cluster is created and it is up and running.

Deploying OLM into the newly created Cluster

I have already mentioned about Operator framework by RedHat, now let's see it in detail.

"The Operator Framework is an open-source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way."

Operator Framework consists of three major tools

Operator-sdk – For the development of the operators.
Operator-Lifecycle-Manger(OLM) – For managing the lifecycle of the operator.
Operator-registry – For storing the operator metadata.

OLM won’t get deployed by default as part of the Kubernetes installation, we should install it separately.

As of now, the latest version released is 0.17.0, by running the below command in the machine which is pointing to the newly created KIND cluster, OLM will get deployed to the cluster.

curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.17.0/install.sh | bash -s v0.17.0

If we do "kubectl get namespace" we can see new two namespaces introduced in the cluster - olm and operators.

By listing the pods in the olm namespace we can see two major pods running in the namespace olm operator and catalog operator.

I know it is getting a little messy now. To install Operator we are installing olm and olm major components are Operators. Crazy right!

So this means to install an operator olm is not mandatory, but for managing the lifecycle of operator OLM can help. OLM has components that will help to upgrade the operator automatically when a new version is available. So no human interruption needed.

OLM Operator

The OLM Operator is responsible for deploying applications defined by CSV(Cluster Service Version).

Catalog Operator

The Catalog Operator is responsible for watching CatalogSources for updates of packages in channels and upgrading them to the latest available versions.

So now the question is what this CSV and CatalogSource.As I mentioned above olm components are operators so it will have custom resource definitions exposed. So CSV, CatalogSources are part of CRD's exposed by OLM operators.

The below diagram shows the list of CRD’s owned by the olm operator and catalog operator.

Deploying Operators using OLM

OK! Now we are ready with the cluster which has olm deployed, so the next thing is we need an operator to deploy in our cluster.So let's visit OpertorHub.

"OperatorHub.io is a new home for the Kubernetes community to share Operators. Find an existing Operator or list your own today. "

In a simple way, I can say Operator hub is like docker hub and it has operator catalogs in it.

For this article, I am going to download and install the ArgoCD operator from the OpertorHub.

Click on the install button and it will show the kubectl command which will install the operator in Kubernetes cluster.

kubectl create -f https://operatorhub.io/install/argocd-operator-helm.yaml

After running the kubectl command, a new namespace created my-argocd-operator-helm and in the namespace, there is an ArgoCD operator pod running.

What happened in Backend, how the ArgoCD Operator got deployed?

Let’s open and see the yaml file we applied to the cluster.
https://operatorhub.io/install/argocd-operator-helm.yaml

First it will create a namespace called my-argocd-operator-helm
Then it will create a OperatorGroup in the namespace my-argocd-operator-helm, The OperatorGroup kind is a CRD exposed by the olm
It will also create a Subscription (CRD exposed by the olm). In Subscription there is a filed to mention the channel. The channel (such as alpha, beta, or stable) helps determine which stream of the operator should be deployed from the CatalogSource.
Subscription also defines from where the catalogs need be downloaded, in our case it is mentioned as operatorhub

When a Subscription is created Catalog Operator will create an Install Plan. By default, OLM will automatically approve updates to an operator as new versions become available via a CatalogSource.
So as mentioned in the subscription catalog operator contact the OperatorHub for the operator argocd
From OperatorHub the latest version from the channel mentioned in the subscription yaml is downloaded and it will have Cluster Service Version.
The olm Operator will install the operator by reading the Cluster Service Version.

This is how the operator got deployed in the namespace my-argocd-operator-helm

Testing with TestProject Docker Agent

Sunish Surendran K — Sat, 28 Nov 2020 08:05:56 +0000

"Well-run tests will help you avoid serious grief in the long run"

The above quote is of great significance as I remember my dark days of software development where I feel like I entered a dark alley without my slingshot or leather sandals.

Of course, I'm referring to proper Testing and avoiding fallacious approaches. But now we came forward from those dark days, the contemporary Testing techniques emerged and it has propelled the need for an efficient, reliable, fast, and powerful automation tool that can fit the diverse skillsets of everyone on an agile team.

Here I am introducing you to one of the newest entrant "TestProject"

An open-source friendly, community-powered automation testing tools. The most amazing part is it has a free forever plan which is fully featured and that you can get started within seconds.

Personally, I have tried TestProject for my research projects. I can strongly argue that it is of high quality and experience so far has been top-notch. So let's dig into it.

💻How to start with TestProject

Step 1 - Signup for free

All we need to do is sign up to Testproject.You can check the documentation if you need help in free signup.

Once you have created an account with TestProject you just need to follow a few simple steps to get setup and ready for testing. TestProject works on almost any platform, with the install of a single agent.

Step 2 - Generate an API key

Generate an API key which allows access to API for your account. In order to do that, open the TestProject application and go to the Integration tab and select the API option.

From there you can create a new API key and set the access you want it to have.Once you've setup that key you can copy it and use it in API call that you make.

🐳 How to setup TestProject Docker Agent

When it comes to the CICD team they have already get rid of the traditional testing and moved to automation. Nowadays, automated testing is embedded so deeply within our perception of software development, it's hard to imagine one without the other. And since that ultimately enables us to produce software quickly without sacrificing quality.

And Docker is the de-facto standard to build, run, test, and share containerized apps from your desktop to the cloud. So for this world TestProject comes with Docker Agent concept. Moreover, the docker agent is useful because it saves a bunch of resources & provides the ability to run tests with just a simple command.

Prerequites

In your machine Docker should be Installed
In this article, we are installing the agent with docker compose yaml file so Docker Compose also should be installed.

Step 1 -Docker-Compose for TestProject Docker Agent

Visit the TestProject dockerhub page, you can see on that page TestProject team explained all the different ways to spin the docker agent.We are using the docker-compose way. The following docker-compose snippet can be used to start a TestProject Agent with headless Chrome & Firefox browsers.

# NOTE: Make sure to update the 'testproject-agent' container volume with a valid local path.
# To execute this docker-compose, store it in a file (e.g. testproject-agent.yaml) an run:
# docker-compose -f <file_name> up -d

version: "3.1"
services:
  testproject-agent:
    image: testproject/agent:latest
    container_name: testproject-agent
    depends_on:
      - chrome
      - firefox
    volumes:
      - .:/var/testproject/agent
    environment:
      TP_AGENT_ALIAS: "Docker Agent"
      TP_API_KEY: "REPLACE_WITH_YOUR_KEY"
      TP_JOB_PARAMS: '"jobParameters" : { "browsers": [ "chrome", "firefox" ] }'
      CHROME: "chrome:4444"
      FIREFOX: "firefox:4444"
  chrome:
    image: selenium/standalone-chrome
    volumes:
      - /dev/shm:/dev/shm
  firefox:
    image: selenium/standalone-firefox
    volumes:
      - /dev/shm:/dev/shm

We need to update the TP_API_KEY & TP_JOB_ID environment variables so that we instruct the Agent to automatically point to our TestProject account and run the job on startup.

In our case since we are not ready with job, so I have not mentioned the TP_JOB_ID environment variable. I have only added the TP_API_KEY that we created.

Step 2-Spinning up the TestProject Docker Agent

I have saved my docker compose yaml file with the name TestProject-DockerAgent.yaml.By running the below command the TestProject agent will be up and running.

docker-compose -f .\TestProject-DockerAgent.yaml up

Within minutes the docker agent is ready with headless Chrome & Firefox browsers. You can see in the below logs saying "Agent initialization is complete"

testproject-agent    | 2020-11-28 14:42:04.491 [INFO ] i.t.a.Program                            *** AGENT - START ***
testproject-agent    | 2020-11-28 14:42:04.742 [INFO ] i.t.a.a                                  TestProject Agent 0.65.10 (64dbf6a7cad45d6e980553444ef4ff002008b3f6) on Linux
testproject-agent    | 2020-11-28 14:42:04.777 [INFO ] i.t.a.a                                  Running as user /home/agent of type Unknown with non-Administrator privileges
testproject-agent    | 2020-11-28 14:42:04.780 [INFO ] i.t.a.a                                  Running inside docker 6a805a004559
testproject-agent    | 2020-11-28 14:42:04.790 [INFO ] i.t.a.a                                  No X11 available - headless mode.
testproject-agent    | 2020-11-28 14:42:04.800 [INFO ] i.t.a.s.IdentityManager                  No identity file was found - Agent is not registered
testproject-agent    | 2020-11-28 14:42:04.804 [INFO ] i.t.a.a                                  Working folder: /opt/testproject/agent
testproject-agent    | 2020-11-28 14:42:04.807 [INFO ] i.t.a.a                                  Data folder: /var/testproject/agent
testproject-agent    | 2020-11-28 14:42:04.816 [INFO ] i.t.a.a                                  Agent IP addresses: 1.1.1.1
testproject-agent    | 2020-11-28 14:42:04.823 [INFO ] i.t.a.m.H                                Checking connectivity with https://testproject.io:443 (Using proxy: No)
testproject-agent    | 2020-11-28 14:42:07.557 [INFO ] i.t.a.m.H                                Connection established successfully.
testproject-agent    | 2020-11-28 14:42:07.560 [INFO ] i.t.a.a                                  Direct connection (no proxy) to TP is possible.
testproject-agent    | 2020-11-28 14:42:08.023 [INFO ] i.t.a.a                                  Agent initialization is complete.

Now we can go back to our TestProject account and check the agent is showing up there. For that, we need to navigate to the Agents Tab. The new docker agent is showing on the page as shown below.

🏃 Running the Test Jobs against the Docker Agent

Under the Projects Tab, I have created a sample test project. You can follow the instruction in the documentation to set up your test.

Let's run My First Test test case against our newly created docker agent by clicking on the play button as shown below

Once you press the play button you will receive a popup and it will ask for which agent to select , in our case we will select the Docker Agent

Once you press the Run button the job will run in the Docker Agent and you can see the logs below

testproject-agent    | 2020-11-28 17:35:13.207 [INFO ] i.t.a.m.c.p                              Successfully handled report for execution KXfcPG0yzUCFyP2Sf24CpQ
testproject-agent    | 2020-11-28 17:35:13.214 [INFO ] i.t.a.m.c.p                              Uploading attachments for execution KXfcPG0yzUCFyP2Sf24CpQ
testproject-agent    | 2020-11-28 17:35:13.217 [INFO ] i.t.a.m.c.p                              Successfully handled attachments for execution KXfcPG0yzUCFyP2Sf24CpQ
testproject-agent    | 2020-11-28 17:35:13.264 [INFO ] i.t.a.m.c.p                              Successfully finished execution KXfcPG0yzUCFyP2Sf24CpQ reporting.

📑 Checking Test Reports

TestProject one of the premium feature that they give us for free
is the report dashboard. To explore Reports we need to navigate to the tab Reports, here we can see the report of our last few runs in the docker agent.

📚 Summary

If you’re looking out for a $0 cloud-based, SaaS test automation development framework designed for your agile team, TestProject is the right choice for you.

"It is manageable, quick and reliable !!!"

As it is built on top of industry-standard open-source tools (Selenium & Appium), supports all major operating systems, and ensures quality with speed using advanced built-in recording capabilities, addons, reports and analytics dashboards, or develop coded tests using TestProject’s powerful SDK for Python/Java/C#!.So in this modern era TestProject suites well!!

Below are few usefull links about TestProject, Check it out !!
Official YouTube Channel

Open Source TestProject's SDK:
Open-source Java SDK | Python SDK | TestProject SDK for Addons

Addons library | Forum