DEV Community: TalhaKhalid101

YAML Based VM Configuration With Cloud-init

TalhaKhalid101 — Tue, 19 Mar 2024 18:37:05 +0000

Cloud-init is a Canonical project and describes itself as follows: " Cloud -init is the industry-standard multi-distribution method for the cross-platform initialization of cloud instances."

To put it more simply: Cloud-init enables the complete configuration of virtual machines (VMs) using a simple text file. Of course, cloud instances mean nothing other than VMs that run on cloud providers such as AWS, Azure or GCC. Basically only with one special feature, namely that the VMs are often fleeting (ephemeral), i.e. they only run for a short time and are then deleted again.

Initialization, in turn, simply means the configuration, i.e. everything apart from the hardware equipment: network connections, settings, installed software, user profiles and so on. And "multi-distribution" probably means nothing other than the fact that you can have several identically configured VMs created at the same time.

Cross-platform means pretty much all major cloud providers are supported; In addition to those already mentioned, Oracle, UpCloud, VMware, OpenNebula, among others, around 25 in total. And there are hardly any restrictions regarding the supported operating systems for the VMs; 26 distributions are listed, in addition to obvious ones such as Debian, Ubuntu , RHEL, Arch Linux and Fedora, also openEuler, Rocky, Virtuozzo and the common BSD systems.

Cloud-init explained in a very rudimentary way: "Dear IT department, we need several VMs with SSH access through Server X, Apache web server, mounted network drives, software packages according to the attached list, the user foobar and a RHEL registration." Cloud-init is nothing more than such a request - just formalized and sent to a supported system.

Cloud-init in Practice

The easiest way to play with cloud-init is probably Multipass , also a Canonical project and already presented here some time ago . With Multipass, "cloud instances" can be created locally, i.e. VMs, which can optionally be set up using cloud-init. This works on Windows as well as Linux.

By default, Multipass creates VMs with Ubuntu 22.04, hardware equipment and network connection are simply specified via options:

multipass launch --mem=2G --cpus=2 --name testvm1 --network name=LAN-Connection

So here an Ubuntu VM is created with 2 GB RAM, 2 CPUs and the (Windows) standard network connection "LAN connection".
If a configuration is to be carried out via cloud-config, the config file is also passed to "multipass launch":

multipass launch --mem=2G --cpus=2 --name testvm1 --network name=LAN-Connection -- cloud-init testconf.ini

All cloud-init work takes place in this "testconf.ini" - a YAML file. YAML is a language based on XML for the machine-readable representation of data structures.

This representation can sometimes be a bit fiddly: be sure to ensure correct indentation. Although it usually doesn't matter whether items are indented with two, four or even three spaces, it must be consistent across the entire document - indentation is part of the YAML syntax! Too many or too few spaces immediately lead to errors.

Cloud-config offers around 60 modules that manage general things like users, but also more specific aspects like the text-based window manager Byobu. A simple version could, for example, simply execute a command in the newly created Multipass VM:

#cloud-config
runcmd:
   - echo „Hello World!"
   - echo „End"

The shebang line is required. The module "runcmd" (run command) is then called, which simply executes the following commands.
Here is a practical example that should do the following tasks:

Output "Hello, World!"
Install the git and apache2 packages
Set up standard user ("ubuntu") and user "peter".
Enable passwordless access from the server "myserver" for user "talha".
Mount a network drive (Samba share) "myserver/myshare".
Write a message to the installation log

Formulated via YAML:

#cloud-config
runcmd:
   - echo 'Hello, World!'
packages:
   - git
   - apache2
users:
   - default
   - peter
ssh_authorized_keys:
   - ssh-rsa AAAAB3... talha@myserver
mounts:
   - [ //192.168.178.1/myshare, /media/myshare, cifs, "defaults,uid=1000,username=talha,password=34565234687gyft@" ]
final_message: |
   cloud-init done
   Version: $version
   Timestamp: $timestamp
   Datasource: $datasource
   Uptime: $uptime

The "packages" module is only used here to install packages, but can also update using the "package_update" and "package_upgrade" calls.

At this point the platform independence is also nice to see. In earlier versions it was called "apt_update", but now the appropriate package manager is targeted. Separate modules are then available for configuring the package managers, e.g. apt, yum/dnf or zypper.

Of course, the "users" module also has a lot more to offer. The full range of account settings can be used here, such as group memberships, user information, passwords, home directories, specific user shell, sudo membership, etc.

But the module "ssh_authorized_keys:" is really interesting. Here it is possible to store the contents of public SSH keys (admittedly in an abbreviated form above) in order to immediately grant password-free access for certain users. This would be practical, for example, for automatic configuration via Ansible , which by default connects to the hosts to be managed via SSH.

The "mounts" module also does exactly what is expected: it writes entries into the fstab - only the separation via comma differs from manual entries.

These examples clearly show what cloud-init essentially does: execute standard commands. Only, for example, it says abstractly "packages: apache2" instead of system-specifically "apt install apache2". You could say that cloud-init is ultimately "just" an abstraction layer for what admins otherwise do manually in the first half hour after setting up a VM… or even on the first day.

More Functions

If you’re interested, it’s best to look through the individual modules yourself; the Cloud-init module reference is very clear and, fortunately, mostly enriched with practical examples. To illustrate the diversity, here are a few highlights.

Using the Ansible module, playbooks can be executed via ansible pull — even without SSH access. There is a module “Byobu” that can be used to configure the window manager mentioned above — very special. The “Phone Home” module is powerful and can be used to send data to a URL.

Similarly powerful is “Scripts per Boot”, which, as expected, executes any script when an instance starts — every time, mind you! Most modules run once per instance, i.e. once during setup. Optionally, they can also be active (“Always”) at every start. For example, “Phone Home” could be used to send data to an API after every start .

Last but not least: Cloud-init also has a command line interface , for example to run individual modules, check cloud-config files, analyze logs and so on.

Cloud-init is a wonderful tool for understanding and implementing “Infrastructure as Code”. And not just on a large scale, by the way, the combination of cloud-init plus Multipass is unbeatable even for local test environments on the desktop.

This article was first published on my medium blog.

10 Tuning Tips to Maximize Your PostgreSQL Performance

TalhaKhalid101 — Thu, 25 Jan 2024 18:59:39 +0000

You’ve decided to use PostgreSQL, one of the most powerful and flexible database management systems available. However, to make the most of this robust feature, it is essential to adjust the settings to meet your project’s specific needs. In this post, we’ll explore some tuning, care, and improvement tips that can help maximize the performance, security, and reliability of your PostgreSQL database and ensure it works efficiently for you.

Choose the right configuration during installation:
When installing PostgreSQL, you will be able to choose the best way to configure your database, for production, approval, web applications, etc. Make sure you perform a configuration that best suits your project needs. There are some websites that can help you with this, for example, PGConfig or pgconfigurator.
Allocate appropriate resources:
PostgreSQL is known to be a resource hog. Therefore, it is crucial to allocate adequate hardware resources, such as CPU, RAM and disk space, and always use your database on a server where there are no other applications competing for resources. Adjust the parameter shared_buffers to allocate a significant amount of RAM to the cache, and track resource usage with tools like pg_stat_statements to make adjustments as needed.
Configure autovacuum:
PostgreSQL uses the autovacuum process to remove old rows and improve its performance. Make sure autovacuum is configured correctly to avoid fragmentation and performance degradation. Adjust the parameters as autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor when required.
Smart Indexes:
Indexes are essential for efficient queries, but excessive use of indexes can degrade performance. Be sure to create indexes only on columns that are frequently used in queries, and remove unnecessary indexes to avoid overhead. Using the table pg_stat_all_indexes can help you analyze which index has been most effective and which is little used.
Constant monitoring:
Use monitoring tools, such as PgBadger or pg_stat_monitor, to monitor the performance of your PostgreSQL. This will help identify potential bottlenecks and issues before they significantly impact your system.
Back up and keep your system up to date:
Remember to make regular backups of your database and keep PostgreSQL up to date with the latest versions and security fixes. It is always recommended to use some tools to perform and manage your backups, for example, pgBackRest, Barman, pg_probackup, etc.
Table Partitioning:
When you deal with large volumes of data, consider using PostgreSQL’s partitioning functionality. This involves splitting large tables into smaller parts, which can significantly speed up queries and reduce maintenance overhead.
Configure Error Logging:
PostgreSQL has a highly configurable error logging system. Be sure to adjust your log settings to record useful information but not overload your system with excessive logging. These logs can help you identify problems and bottlenecks in database queries.
Tuning Operating System Parameters:
In addition to tuning PostgreSQL parameters, be sure to optimize your operating system settings for PostgreSQL. This may involve configuring resource limits, such as file handles, according to the needs of the database.
Horizontal Scalability with Replication:
When growth is a concern, replication is an important option. Configure PostgreSQL replication to distribute load between servers and ensure high availability. To do this, you can use tools such as pgPool, HAProxy or even do this within your own application.

PostgreSQL is certainly an excellent and versatile choice for your database projects, but success largely depends on how you tune and optimize its settings. With these tuning tips, you can ensure your PostgreSQL runs at maximum efficiency and performance, regardless of the size or complexity of your project.

Always remember that PostgreSQL tuning is an ongoing process. As the volume of data or number of users increases, you may need to re-evaluate and adjust settings to maintain optimal performance. Stay up to date with best practices and continue to improve your PostgreSQL database to meet the needs of your application or system. With the tips above, you are on your way to achieving highly optimized PostgreSQL.

This post was first published on my medium blog.

Call a SOAP service from Postman or Fiddler Using Azure Web Apps

TalhaKhalid101 — Tue, 25 Jul 2023 15:53:09 +0000

While it’s true that JSON or XML has been the rage for a long time in web services, there are still thousands and thousands of SOAP services in most companies I walk across.

Today I had to do some tests with a super simple example with WCF (in fact, the one that comes with the Visual Studio WCF template) and the truth is that I spent a while until I was able to access the service from Postman or Fiddler correctly. The fact is that I have tried with WCF Test client, which comes as part of Visual Studio and it worked without problems.

But when I got to Postman, all I got was a 400 — Bad Request error without much explanation.

And in Fiddler exactly the same!

After wandering around for a while, I uploaded my service to a Web App in Microsoft Azure and tried one of the many pages that act as a SOAP client, reading from your service definition (WSDL), and that’s where I was able to see what was missing from my request to make it work just as well as in WCF Test Client: the SOAPAction header, plus a correct Content-Length in the case of Postman. In such a way that the configuration to be able to make the call would be the following:

These would be the headers needed to make the request:

and this is the body of the message:

For Fiddler they would be the same values but in their format:

An additional point to review is that Postman does not calculate the Content-Length automatically as Fiddler does and this can also cause problems when making the request. You can use a page like this (or Fiddler that calculates the content length and updates it by itself) to calculate the size of the content and save you a few more minutes.

Kubernetes 1.24 Released: What’s New?

TalhaKhalid101 — Wed, 11 May 2022 21:01:30 +0000

The latest Kubernetes version 1.24 “Stargazer” brings a lot of changes to the open-source system for automating the deployment, scaling, and management of containerized applications. 14 features are now permanently available after the test phase, 15 innovations are in the beta phase and 13 improvements are being tested in the alpha phase. In addition, two features were deprecated and two others were permanently deleted. The main topics of this release include:

Stargazer

The Kubernetes development team came up with a special name for version 1.24: Stargazer. According to its own statement, this name is intended to refer to the fact that gazing at the stars can spark scientific inquiry and creativity and facilitate navigation in unknown areas. The logo is also imaginatively designed: it shows a stylized telescope in front of a starry sky. The constellation shown is the Pleiades, also known as the seven sisters in Greek mythology. It pays homage to the original internal project name of Kubernetes: Project 7.

Dockershim removed from kubelet

Dockershim was the CRI-compliant layer between Kubelet and the Docker daemon. After being deprecated in version 1.20, the Dockershim component has now been permanently deleted from the kubelet.

As of version 1.24, either one of the other supported runtimes (e.g. containerd or CRI-O) or, in case you still want to rely on Docker Engine, cri-dockerd must be used. Further information on precautions that may be necessary due to the removal of Dockershim is provided by Kubernetes in a guide.

New beta APIs are no longer active by default

New beta APIs are no longer enabled by default in Kubernetes. However, existing beta APIs and the new versions of existing beta APIs will still be on board by default. More on this topic can be found on GitHub.

Contextual logging in the alpha phase

The Contextual Logging feature was introduced with Kubernetes version 1.24. It allows a function’s caller to control all aspects of logging, such as output formatting, verbosity, additional values, and names.

Additional Information

Information on further updates can be found on the Kubernetes website. The full release notes can be viewed on GitHub. The team behind the release of Kubernetes 1.24 will host a release webinar on May 24, 2022, from 9:45 am to 11:00 am Pacific Time. Registration options and the program can be found on the event website.

Kubernetes 1.24 is available for download on GitHub. There is also an interactive tutorial for getting started with Kubernetes. All release information is available in the Kubernetes Release Announcement.

If you liked this blog post, consider following me for more interesting guides and tutorials.🙂 Also, don’t forget to buy me a coffee right here.

5 Cool React Libraries You Should Know [not the usual one's]

TalhaKhalid101 — Fri, 19 Nov 2021 20:21:03 +0000

After several weeks of writings about Kubernetes, I want to write a short post about React, about libraries, because man can not live only on Kubernetes.

These are some of the libraries that I consider most useful and cool in React. For obvious reasons, React-router, Redux, and other well-known ones are excluded. As well as some React Frameworks such as Gatsby, Nextjs, Frontity, and others.

Ant Design

Ant design is beautiful, yes, there is not much that can be deepened using text. It has tons of components that are visually pleasing and very stylish: buttons, sliders, progress bars, layouts, you know, the basics. Make sure to visit their site and see for yourself all that Ant design has to offer when you are done reading this post, of course.

Formik

Formik is a great library which makes working with forms simple and scalable. It allows you to have controlled fields, create validations, reset the form, set a status, handle errors, all with a few lines of code: we define an object that contains properties with their respective validations and voila, formik takes care of almost everything.

Note the validation schema on the left side consisting of an object called ValidationSchema which has the name of the fields and functions that are concatenated to carry out the validation. There are functions like min _(), max (), oneOf () _, and many others for almost any type of validation you require. I leave you the link to the sandbox from where I took this example.

React query

Every time an API request is made, there is code that is repeated; make the request, display an item indicating that content is loading, receive the error or successful status, and save it to the status. Does it sound familiar to you?
React query is responsible for reducing all the repetitive code that is responsible for the entire process of handling web requests, providing us with a special hook from which we can unstructured variables that will facilitate the handling of the response.

React-icons-kit

Sometimes it is quite annoying to take care of the graphic part of a web page. There are icons everywhere but you have to look for them, sometimes an icon pack does not have all the icons we need and we have to combine different ones. An excellent solution to these problems is React-icons-kit.

Before using it, remember to check the license of the icons you decide to use, because not all licenses are equally permissive.

The minimalist React: Preact

Preact is React, yes, same functions, well, not all actually, but the most common ones yes, all in just 3kb. Preact promises to be much faster and lighter than its counterpart as it uses the browser's native addEventListener instead of React's synthetic event handler. It also has exclusive functions that you can't find in React. This library is ideal for applications where performance is a critical factor.

You can read more differences between React and Preact on their official page.
Here's a Bonus!

React Virtualized

React virtualized takes care of solving a problem that seems pretty simple at first. Render lists and information that can be tabulated. Only that? Well yes, but rendering lists with a few items wouldn't be a problem, would it? The strength of React Virtualized is not rendering small lists, but large lists, greater than 1k of elements with most of the problems that are already solved and tested.

Visit the React Virtualized page to read the full documentation.

If you liked this blog post, consider following me for more such stuff. Also, feel free to add your thoughts!🙂Or you can buy me a coffee right here.

Understand unit tests once and for all [In less than 3 min]

TalhaKhalid101 — Sat, 13 Nov 2021 22:37:20 +0000

One day, browsing through software development communities I came across the following image:

I confess that the first time I didn't understand what the image was about, but I was sure that a cheap rabbit was better than an expensive turtle.

Time passed, I finally learned what unit tests are, what this cheap rabbit represents and I would like to help you understand too.

But what are unit tests?

A unit test is basically testing the smallest testable part of a program.

Okay, but what does that mean?

If you pr the grass in a language that supports functional paradigm for example, the smallest testable part of your code should be a function. So a unit test would be the test of any function. In the case of object orientation it would be the testing of a method of your object.

Well… as everything is easier to understand in practice, let's go to the code!

In the example below, we have a function that adds two numbers and returns the sum value.

function sum(a, b){ return a + b; }

To test this code, all we need to do is run the function and check if its output value is what we expect.

var result = sum(1, 2); expect(result).to.equal(3);

Ready! We already have our first unit test. Pretty easy isn't it?
Well.. Now that you understand what unit tests are and how unit tests are done, you might be asking yourself:

What are unit tests for?
Asking what unit tests or any other automated tests are for is a great question, after all there are several ways that apparently are faster to test if my function is doing what it should. I could just run the code to make sure it's working.

So why am I going to write other code to test my code? What makes sure the second code works? Who tests the test?

Unit tests, as well as any automated tests, are not primarily for verifying that a specific function is working, but rather to ensure that your application continues to work after any changes to your code base.

Why write unit tests?
It may seem tempting at first not to write tests for a function you have just developed, after all, it is customary to write more code to test a function than the function code itself. But you must remember that you will spend most of your development time on a system working on its maintenance.

Your application will soon have a few hundred functions running, and in many times running each other, your code base gets huge and soon becomes humanly impossible to manually test after any change. Unit tests most often take just a few seconds to test your entire application.

Where to start?
There are several unit testing tools for each programming language. You can get started by reading the documentation for these tools from their examples.

Okay, but I still don't understand the cheap Rabbit and Expensive Turtle thing
I won't close this post without first explaining what the test pyramid means.

It's a concept developed by Mike Cohn that says you should have a lot more unit tests (unit tests) than GUI tests which are more user-level tests. Like clicking on a link for example.

The Pyramid Concept explains how costly GUI tests are compared to unit tests, as they take much longer to run and are also difficult to maintain, while unit tests are much simpler, faster and cheaper.

If you liked this blog post, consider following me for more such stuff. Also, feel free to add your thoughts!

Diagnosing Why Your Kubernetes Pod Won’t Start

TalhaKhalid101 — Wed, 10 Nov 2021 15:10:55 +0000

A pod is a minimum and basic unit that is handled by Kubernetes. The pod consists of one or more containers that are connected to each other and share storage yet run independently. An application-specific pod model contains one or more containers that go on the same machine.

Kubernetes manages things like automatically restarting a pod when it fails. However, it sometimes happens that the pod does not start when you create it for the first time or update it with new configurations. This can happen for a variety of reasons. So,today, I'll briefly look at why this happens!

Header Reasons Your Kubernetes Pod Won’t Start

A pod may not start or fail due to various reasons, some of which are:

Pod Status is Pending
The pending status implies that the pod’s YAML file has been submitted to Kubernetes, and an API object has been created and saved. However, Kubernetes could not create some of the containers in this pod. A scheduling conflict resulted in the situation not working out. Using kubect to describe the pod, you can see the current pod event and determine why it does not have a schedule.

Here are a few possibilities:

The cluster nodes cannot meet the CPU, memory, GPU, and other resource requirements requested by the pod.
The service port needs to be opened to the outside by using HostPort, but it is already busy.

The Pod is in Waiting or ContainerCreating State

This can occur due to one of the following issues:

There is an incorrectly set image address, a foreign image source that cannot be accessed, an incorrectly entered private image key, or a large image that causes the pull to time out.
Errors of this kind may result from a problem with the CNI network plug-in settings, such as an IP address can’t be assigned or the pod network cannot be configured.

Make sure that the image is packaged correctly and that the container parameter settings are correct. When the container doesn't start, make sure that the image is packaged correctly.

The Kubelet log should show the cause of the error. Usually, it comes from a failed disk (input/output error) that prevents pod sandboxes from being created.

ImagePullBackOff Error

Generally, the ImagePullBackOff error occurs if the private image key configuration is incorrect. In this case, you can use docker pull to verify that the image can be extracted normally.

If the private image key is set incorrectly or not set, then:

Check the docker registry type secret

# Ver Docker-Registry Secret $ kubectl get secrets my-secret -o yaml | grep 'dockerconfigjson:' | awk '{print $NF}' | base64 -d

Set up a secret of type docker-Registry

# First create a docker record type Secret $ kubectl create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

CrashLoopBackOff Error

The CrashLoopBackOff error occurs when a pod is running, but one of its containers restarts due to termination. This basically means that the pod was not able to run and crashed, so Kubernetes restarted it. Unfortunately, thereafter the pod crashed again and was restarted again, forming an endless loop.

This can be caused by a deployment error, aliveness probe misconfiguration, or a server-init configuration error.

By configuring and deploying Kubernetes correctly, you can solve this error quickly.
Using a blocking command, you can bypass the error and create an entirely separate deployment.
Any method that involves back-off restarting failed container should help resolve this error as well.

Pod Is in Error State
In most cases, this error status indicates that the pod is having an error during the startup process. Common causes of the error state include:

dependent ConfigMap, Secret, or PVWaiting does not exist: The requested resource exceeds the limit established by the administrator, such as exceeding LimitRange.
There is a cluster security policy violation, such as the violation PodSecurityPolicy, or the container does not have the right to operate the resources in the cluster, such as open RBACA.

The Pod Is in a Terminated or Unknown State
When a node fails, Kubernetes does not remove the pods on its own but marks them as terminated or unknown. The following three methods can be used to remove pods from these states:

Take the node off the cluster: If a VM is removed after the kube-controller-manager has been installed, the corresponding node will be removed automatically.
A physical machine node must be manually removed (using kubectl remove node) in a cluster installed on physical machines.
Node activity resumes normal: To determine whether these pods are to be removed or made to continue running, Kubelet will contact Kube-apiserver from time to time to confirm the expected status of these pods. podSpec parameters are set within. This is usually because the podSpec YAML file content is wrong: you can try using validate parameters to rebuild the container.

Conclusion
A Kubernetes pod can fail due to improper configuration or due to a problem in the code. Quite often, it’s the former, and if you know how to detect the error, resolving it becomes much easier.