DEV Community: Lakshan Dissanayake

OpenTelemetry

Lakshan Dissanayake — Thu, 20 Jun 2024 19:06:40 +0000

What is OpenTelemetry?

OpenTelemetry is an open-source project that provides a set of APIs, libraries, agents, and instrumentation to enable observability in software applications. Tracing is one of the core features of OpenTelemetry, which helps in tracking the execution of operations within and across services.

What is OpenTelemetry Tracing?

OpenTelemetry tracing is a way to capture and visualise the flow of requests as they traverse through different components of a distributed system. It allows developers to understand how different parts of an application interact and identify performance bottlenecks, errors, and latency issues.

Key Concepts

Trace

A trace represents the entire journey of a request as it moves through a system. It consists of a series of spans that are linked together.
Span

A span represents a single operation within a trace. It contains information such as the operation name, start and end timestamps, attributes (metadata), and references to other spans (parent/child relationships).
Context Propagation

Context propagation allows trace context to be passed along with requests as they move through different services and components. This ensures that spans are correctly linked together to form a complete trace.
Sampling

Sampling determines which traces are collected and reported. It helps control the volume of trace data, balancing the need for observability with resource constraints.

How OpenTelemetry Tracing Works

Instrumentation

OpenTelemetry provides instrumentation libraries for various programming languages and frameworks. These libraries automatically capture trace data for common operations like HTTP requests, database queries, and more.
Context Injection

When a request enters a service, OpenTelemetry injects trace context into the request headers. This context is propagated through downstream services, ensuring that all operations related to the initial request are included in the same trace.
Span Creation

Each operation within a service creates a new span. Developers can create custom spans to capture additional operations or specific code blocks.
Trace Exporting

Collected trace data is exported to a backend for analysis. OpenTelemetry supports various backends, including Jaeger, Zipkin, Prometheus, and more. The exported data can be visualised to understand request flows and identify issues.

Benefits of OpenTelemetry Tracing

End-to-End Visibility

Provides a comprehensive view of how requests are processed across different services, helping to identify bottlenecks and improve performance.
Error Detection

Helps in identifying and diagnosing errors by showing where they occur in the request flow.
Performance Optimisation

By measuring the latency of different operations, developers can pinpoint and optimise slow parts of the system.
Improved Debugging

Detailed trace data makes it easier to debug complex issues that involve multiple services.

golang unit testing

Lakshan Dissanayake — Sat, 03 Sep 2022 17:14:51 +0000

Introduction

Test driven development also known as TDD is a good software development practice followed by all over the software industry. With unit tests we can test a small pieces of our software independently what makes it "unit".

Problem

Let's assume you're making some changes to a function which was developed by someone else previously. with the developer test you might think, it is safe to deploy. But what will happen if you missed a specific test scenario? probably it will lead to a software crash if exceptions are not properly handled. panics in go's context.

Solution

If we have already developed unit tests, then we can run it to test whether we broke the code or not. It will execute a list of specific scenarios of the software and shows if some flows are broken. if there is not unit test, I highly recommend you to implement it first, then do the changes to the function.

Libraries we can use to write unit tests

I use gomock or mockery for mocking the interfaces and testify for evaluating tests

Let'a have a look how its implemented

Sms Provider

Email Provider

And suppose notification service, which validates phone numbers and email and calls sms provider and email provider to send actual email and test messages

The unit tests of the notification service can be written as the following

golang multithreading

Lakshan Dissanayake — Mon, 02 May 2022 20:07:57 +0000

In my workplace, client had reported a performance bottle-neck and as a resolution, I was assigned to identify the bottleneck and make necessary changes to make it more efficient. After some time investing the logs, I had found the root cause and it was due to the repetitive occurrence of some time-consuming activity, which ultimately impacted on the response time.

After some workaround, I decided to execute the time consuming operations asynchronously and wait for the execution of asynchronous work and aggregate the final results. Here I found 3 ways to do this.

go-routines are light weight wrappers for threads, they will do the work using a pool of threads internally.

The Bad way

TLDR: Execute jobs synchronously.

This is a bad way of doing asynchronous work since it blocks main thread and execute jobs synchronously.

The Normal way

TLDR: Execute jobs asynchronously, but number of parallel jobs are uncontrolled

In here, sync.WaitGroup is controlling the execution of background thread and it waits till the atomic counter of waitGroup is reduced to zero, which indicates all the routines have been executed. The problem with this approach is, if we get a large number of jobs in job array, there will be that number of go-routines which consumes resources.

The Better Way

TLDR: Execute jobs asynchronously and number of parallel jobs are controlled

This approach will do the work in a controlled parallel manner. So at any given time, it will use max number of go-routines to execute the jobs parallelly and will consumes less resources.

implementing retry mechanism in rmq

Lakshan Dissanayake — Fri, 29 Apr 2022 12:13:06 +0000

Rabbit MQ is an opensource message broker. We can publish messages to exchanges and queues can be bound to exchange with routing keys. Consumers will consume the payloads from the queues and so the application will work as intended.

Lets assume for some reason application couldn't process the payload and it rejects it, and we need to retry it with some delay? This is not rocket science, but lemme show you how I would have done it.

Pre-Conditions

q_work is bounded to ex_main with rk_work
DLX of q_work is ex_retry
DLX Routing Key of q_work is work
q_retry_work is bounded to ex_retry with rk_work
DLX of q_retry_work is ex_main
DLX Routing Key of q_retry_work is work
TTL of the q_retry_work is 30s

Producers publishes messages to ex_main.
Messages with rk_work routing key is immediately pushed to q_work.
The application will consume or reject messages.
Rejected messages are dropped to ex_retry and messages with rk_work will immediately pushed to q_retry_work.
Since no one is consuming from q_retry_work and after 30s it will dropped the messages to its DLX, which is ex_main.

Pipenv and Django

Lakshan Dissanayake — Tue, 26 Apr 2022 15:17:39 +0000

How to setup pipenv for django

pipenv is a tool that allows you to create virtual environments and manage python dependencies effectively. If you need more information regarding pipenv, Read this

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source. Ridiculously fast.

#setup
# create solution directory
$ mkdir my-first-django-project 
$ cd my-first-django-project
# install pipenv
$ pip install pipenv
# installs django framework
$ pipenv install django
# create django project
$ pipenv run django-admin startproject project .
# initialize django app
$ pipenv run django-admin startapp app
# fire-up python web browser
$ pipenv run python manage.py runserver

Server monitoring with Zabbix

Lakshan Dissanayake — Tue, 26 Apr 2022 15:13:31 +0000

We were developing a shopping cart application for a client using WordPress. During the development, we faced an occasional server not responding situation. So we wanted to monitor the availability of the Linux server.

After several hours of searching and exploring, we found several server monitoring solutions.

Zabbix was the obvious choice for us as we were seeking for an open-source monitoring solution. It was quite easy and could able to get online pretty soon. Just go through the installation instructions of Zabbix and get your servers monitored.

Golang Job Queue Implementation

Lakshan Dissanayake — Tue, 26 Apr 2022 15:01:50 +0000

Introduction

A friend of mine asked me to implement a job-queue. As I was new to Golang at that time I was curious and talked with the friend and also browsed the internet and read the purpose of a job queue.

Eliminate performance bottlenecks

During the research time, I found using JSON is causing a huge impact on performance. The solution was to use gRPC with Protobuf. I used a channel with size with several thousand as the main queue. Since there is limited no of worker machines, I needed to utilize the processor with maximum efficiency to get the jobs done quickly. So each worker machine takes job parameters from the main queue and stores them in their own queue which allows making the room for new requests in the main queue.

These worker machines are equipped with several Goroutines. Each Goroutine takes job parameters from its own worker machine and executes the job.
Even though Goroutines are extremely light-weight thread like programs, using millions of Goroutines won’t result in the optimal utilisation of resources. Please read this article.

My approach to the job-queue implementation
The central server application listens to client requests and put them into the main queue. Several worker machines take job parameters from the main queue and store them on their own queue. Each worker machine is equipped with several small processes whose sole task is to execute the job parameters which are extracted from the queue in the worker machine.

Artifacts

Let the no of worker machines to be m and the no of Goroutines each worker machine runs to be n, on maximum resource utilisation, the applications execute m x n concurrent jobs at any given time.

Source Code

https://gitlab.com/lakshanwd/go-job-queue

Configure CentOS to host WordPress with Nginx

Lakshan Dissanayake — Wed, 04 Nov 2020 10:13:08 +0000

I’ve been told to set up a WordPress installation on a CentOS server. While most of CentOS based WordPress hosting providers offer cPanel, I had thoughts of writing a tutorial regarding configuring a server from zero.

A WordPress site consists of 3 main parts

PHP Interpreter
Database
Web Server

Let’s walk through the tutorial and find out what each of these does.

Setup MariaDB

Even though WordPress is written in PHP, it needs a MySQL/MariaDB database for its functionality. So we’ll choose MariaDB as our SQL database.

Install MariaDB

CentOS is packed with an old version of MariaDB which impacts on overall performance. So let’s update the MariaDB source repository configurations and install it.

# install mariadb-server
wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
chmod +x mariadb_repo_setup 
sudo ./mariadb_repo_setup
sudo yum install mariadb-server

Configure MariaDB

You need a database and properly configured MySQL user for the access of the database. let’s create a database and user.

Even though you can use MySQL root for WordPress, I strongly discourage you to do so. Because if somehow your password gets compromised, You’ll compromise other databases on your host as well.

# configure mariadb to start automatically upon system boot
sudo systemctl enable mariadb.service
sudo systemctl restart mariadb.service
# enhance security by setting up root password and other enhancements
sudo mysql_secure_installation
# login to mysql
sudo mysql -u root
# create database
mysql > CREATE DATABASE wordpress DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;
# create user
mysql > grant all privileges on wordpress.* to 'wp_user'@'localhost' identified by 'secure-password';
mysql > flush privileges;

Setup PHP and Nginx

We need a PHP interpreter to render raw HTML from PHP scripts and we also need a webserver to serve those. Since we’re seeking for an open-source server solution with robust speed there are several server solutions. For the purpose of this demonstration, we will Nginx as the webserver.

Install Nginx and PHP

By default, Nginx is not listed in CentOS software repository and also the PHP version it bears is an old one. So let’s update the software repositories and install Nginx and PHP with other dependencies.

# setup required repositories for Nginx/PHP
sudo yum install -y epel-release
# setup required repositories for php74-php-fpm
sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum -y install https://rpms.remirepo.net/enterprise/remi-release-7.rpm
# install Nginx
sudo yum install -y nginx
# install PHP
sudo yum install -y php74-php-fpm php74-php-mysqlnd php74-php-mbstring php74-php-zip php74-php-gd php74-php-imagick php74-php-xml
# enable and restart nginx php74-php-fpm services
sudo systemctl enable nginx php74-php-fpm
sudo systemctl restart nginx php74-php-fpm

PHP and Nginx are been installed successfully at this point. But we need to change some configurations of PHP and Nginx as well.

Configure PHP

Edit the /etc/opt/remi/php74/php.ini and make the following changes

# uncomment cgi.fix_pathinfo and change the value to 0
cgi.fix_pathinfo = 0
# change the value of post_max_size
post_max_size = 64M
# change the value of upload_max_filesize
upload_max_filesize = 32M

Edit the /etc/opt/remi/php74/php-fpm.d/www.conf as follows.

# change  the user to nginx
user = nginx
# change  the group to nginx
group = nginx
# uncomment and change  the listen.owner to nginx
listen.owner = nginx
# uncomment and change  the listen.group to nginx
listen.group = nginx
# change listen as follows
listen = /run/php74-fpm.sock

Download WordPress

Let’s take a fresh build of WordPress from the WordPress site and extract it where you want your site to be located.

# download latest WordPress build
wget https://wordpress.org/latest.tar.gz
sudo tar -xzf latest.tar.gz -C /path/to/web-site

Change the ownership of extracted WordPress files
Changing the ownership of the WordPress files to the PHP user is an absolute requirement. Otherwise, PHP interpreter won’t be able to access those files and you face issues with rendering PHP.

# change the owner of the website to nginx
sudo chown -R nginx:nginx /path/to/web-site

Set SELinux permissions

Since SELinux bundled with CentOS, We’ll have to let SELinux knows that the extracted files will be read and written. Execute the following to do so.

sudo setenforce 1
# enable content readable by php-fpm
sudo chcon -Rt httpd_sys_content_t /path/to/web-site
#enable content writable by php-fpm
sudo chcon -Rt httpd_sys_rw_content_t /path/to/web-site/wrirable-dir

Configure Nginx

Create the following directories

# holds all (active / inactive) site configurations
sudo mkdir -p /etc/nginx/sites-available
# holds Nginx active site configurations
sudo mkdir -p /etc/nginx/sites-enabled

Edit /etc/nginx/nginx.conf and put the following content at the end of http block

# enable gzip compression
gzip              on;
gzip_buffers      16 8k;
gzip_comp_level   5;
gzip_http_version 1.0;
gzip_min_length   10240;
gzip_types        text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon image/bmp;
gzip_vary         on;
gzip_proxied expired no-cache no-store private auth;
gzip_disable "MSIE [1-6]\.";
# per site configurations
include /etc/nginx/sites-enabled/*;

Create an empty file in /etc/nginx/sites-available/my.domain.tld and put the following content

server {
    server_name my.domain.tld;
    root /path/to/web-site;
    index index.php;
    error_log /var/log/nginx/my.domain.tld/error.log;
    access_log /var/log/nginx/my.domain.tld/access.log;
    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }
    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }
    location / {
        try_files $uri $uri/ /index.php?$args;
    }
    location ~ \.php$ {
        include fastcgi.conf;
        fastcgi_intercept_errors on;
        fastcgi_pass unix:/run/php74-fpm.sock;
    }
    location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
        expires max;
        log_not_found off;
    }
    client_max_body_size 64M;
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/ssl/my.domain.tld-certs/certificate.pem;
    ssl_certificate_key /etc/ssl/my.domain.tld-certs/privatekey.pem;
}
server {
    if ($host = my.domain.tld) {
        return 301 https://$host$request_uri;
    }
    server_name my.domain.tld;
    listen 80;
    listen [::]:80;
    return 404;
}

Link the configuration to Nginx

# activate the site
sudo ln -s /etc/nginx/sites-available/my.domain.tld /etc/nginx/sites-enabled

Reload Nginx and PHP

Let’s restart the Nginx and PHP services.

# restart nginx and php-fpm
sudo systemctl restart php74-php-fpm nginx

Congratulations

Yay!!! You have successfully configured a CentOS server with PHP and Nginx. There are a lot more configurations that we didn’t even talk about in this tutorial. Just browse and look at the official documentation of Nginx, PHP and SELinux.

Don’t forget to strength up your server by implementing a Firewall. So if you have done everything correctly, with a proper DNS direction, your site should be online by now.

DEV Community: Lakshan Dissanayake

OpenTelemetry

What is OpenTelemetry?

What is OpenTelemetry Tracing?

Key Concepts

Trace

Span

Context Propagation

Sampling

How OpenTelemetry Tracing Works

Instrumentation

Context Injection

Span Creation

Trace Exporting

Benefits of OpenTelemetry Tracing

End-to-End Visibility

Error Detection

Performance Optimisation

Improved Debugging

golang unit testing

Introduction

Problem

Solution

Libraries we can use to write unit tests

golang multithreading

The Bad way

The Normal way

The Better Way

implementing retry mechanism in rmq

Pre-Conditions

Pipenv and Django

How to setup pipenv for django

Server monitoring with Zabbix

Golang Job Queue Implementation

Introduction

Eliminate performance bottlenecks

Artifacts

Source Code

Configure CentOS to host WordPress with Nginx

Setup MariaDB

Install MariaDB

Configure MariaDB

Setup PHP and Nginx

Install Nginx and PHP

Configure PHP

Download WordPress

Set SELinux permissions

Configure Nginx

Reload Nginx and PHP

Congratulations