DEV Community: Matt Ghafouri

Observability or Monitoring: Which one do you need?

Matt Ghafouri — Fri, 16 Feb 2024 11:34:57 +0000

Observability or Monitoring: Which one do you need?

As software engineers, we have most likely heard a lot about monitoring, but what about observability? what is the difference between them? Why do we need them in the first place? These are the topics that we are going to discuss in this article.

Table of Contents

Monitoring Definition
Observability Definition
Monitoring vs Observability
Observability main components
Visualization and Dashboard
Incident Automation
Challenges and Pitfalls
Benchmarking Observability Tools

You can also watch this article’s video on YouTube

What is Monitoring?

🔖Short Definition

Monitoring involves the systematic collection of data about a system’s health, performance, and other key metrics.

🔖Long Definition
In a broader sense, monitoring entails identifying specific areas within the application that require attention. By strategically placing logs in these identified locations, we gain the ability to observe and assess the system’s behavior. The essential aspect of monitoring lies in the proactive observation of anticipated issues, with an expectation that the system will manifest expected behaviors in those instances.

🔖Purpose
Detect and alert on deviations from expected behavior.

🔖Examples

Monitoring the CPU usage
Tracking the response time of a web application
Logging errors or exceptions in an application for later analysis

🔖Tools

What Is Observability?

🔖Short Definition
The measure of how well you can understand the internal state of a system based on its external outputs.

🔖Long Definition
Observability encompasses a broader spectrum of application performance, extending beyond internal status. Its primary focus lies in addressing the larger picture of the entire system rather than concentrating on a single service.

Its significance becomes evident when applied to complex systems that require monitoring from various aspects, such as the relationships between services, internal service statuses, communication statuses, and more.

In essence, during debugging and troubleshooting, observability empowers developers or system administrators to comprehensively assess the system’s overall status by scrutinizing each component.

Observability platforms prove especially valuable when dealing with unknown issues, like incidents claiming a low response time for the order service. In such cases, these platforms come into play, facilitating the examination of all services and their interrelationships.

It’s not just about reading logs, as system performance issues may not always manifest as errors in the logs, but rather as improper system functioning.

🔖Purpose
Provide insights into the system’s internal workings and facilitate debugging and troubleshooting.

🔖Examples

Instrumenting code to capture detailed traces, logs, and events.
Using distributed tracing to follow the flow of a request.
Collecting and analyzing logs, and metrics.

🔖Tools

Dynatrace
DataDog
Splunk
Logz
SumoLogic
Cloud Observability tools (AWS, AZURE, GCP)

Key Components of Observability

Logs: Including system and server logs, network system logs, and application logs
Metrics:Monitoring CPU and memory usage, infrastructure metrics
Traces:Track the performance of microservices

Monitoring vs Observability

Monitoring and observability can be differentiated based on various perspectives, with the primary ones being :

Focus , Purpose, Data Collection, Alerting, Use Case, Scope, Tools

Data Collection and Instrumentation

Data collection and instrumentation are integral components of observability, providing the means to gather information about the internal workings and performance of a software system.

Data Collection

🔖Definition
Involves gathering relevant information and metrics from various components within a software system.

🔖Purpose
The collected data helps in monitoring, analyzing, and understanding the system’s behavior, performance, and health.

🔖Example
In this image, depicting a configuration file associated with the Prometheus Monitoring Platform, our objective is to extract metrics from an application hosted at the address localhost:5000. The scraping interval set for this operation is 15 seconds. This signifies that the Prometheus service will, at regular 15-second intervals, retrieve metrics such as CPU usage, Memory Usage, and more from the specified endpoint.

Later on, we’ll delve into the internal components of the Prometheus server, exploring how metrics are collected and persistently stored.

Instrumentation

🔖Definition
Involves adding code or agents to a software system to gather specific data and metrics at runtime.

🔖Purpose
Instrumentation provides a way to gather fine-grained insights into the behavior of the application, allowing for detailed analysis and troubleshooting.

🔖Example
Within this code snippet, utilizing the Prometheus SDK, we can explicitly gather specific metrics within our application. For example, in this code, as we receive a request, the request counter increments by one unit. This framework allows you to expose various types of metrics essential for monitoring your application’s performance and health by instrumenting your code.

To expose your custom metrics, all you need to do is configure the Prometheus service, already hosted on your service. Utilizing your specific programming language SDK, you can then expose the desired metrics from within your code.

Data Quality Matters

To maintain the quality of collected data, data sources from various monitoring systems should be standardized to prevent redundancy, reduce clutter, and minimize noise.

In a large and intricate system, we gather diverse metrics such as application heartbeat, application logs (Information, Debug, Exception, etc.), as well as metrics related to application performance and resource usage.

Given the substantial volume of data collected, it becomes crucial to implement a strategy for filtering only the essential metrics.

One effective approach involves sampling metrics in conjunction with the Telemetry Framework, enabling us to selectively filter out necessary metrics. For example, by sampling metrics associated with server responses having a status code greater than 300, we specifically focus on gathering responses that indicate not successful, excluding those categorized as successful.

Visualization and Dashboards

Providing a user-friendly interface to analyze and interpret complex data from diverse sources.

Once we’ve collected metrics, the next step is to derive insights from them. However, each team may have unique requirements, necessitating their specialized dashboards and graphs.

Some teams may require real-time data on application health status, while others may need information on the rate of messages received from a Kafka topic, and so forth. Consequently, the selection of an observability platform capable of accommodating these varied needs becomes crucial when adopting an observability platform.

🔖 Grafanaspecializes in customizing dashboards and visualizing metrics.

🔖 Kibana, when paired with the ELK stack, facilitates the collection and querying of logs through a user-friendly dashboard.

🔖 Dynatrace, on the other hand, stands out as an enterprise-level observability tool, encompassing all the necessary features for visualization and dashboard creation.

Framework for data collection

These frameworks are used to instrument code, capture telemetry, and enable monitoring, tracing, and logging.

Having discussed code instrumentation earlier, it’s important to note that Prometheus is not the sole framework available for this purpose. There are several other notable frameworks for data collection and code instrumentation.

Prometheus Overview

Exploring how the Prometheus server operates and facilitates the metric collection, notification to other services, and exposure and visualization of collected metrics reveals three main components at its core:

Retrieval: This component enables the collection of metrics from various applications, short-lived jobs, and services. Operating on a pulling mechanism, it retrieves a list of targets from service discovery.
TSDB(Time-series Database): Prometheus provides a Time-series database that can be hosted on a separate node. It supports vertical scaling and federation for scenarios requiring multiple instances of the database.
HTTP Server: Prometheus includes an endpoint through which it exposes its metrics to the external environment.

Additionally, Prometheus incorporates the following components to enhance its functionality:

Push Gateway:Some jobs and services may be unable to directly expose or push their metrics externally. The Push Gateway allows Prometheus to pull metrics from these services and jobs.
Alert Manager: This component enables Prometheus to send various types of notifications (Email, SMS, Slack, Microsoft Team, etc.) to relevant teams, enhancing communication and alerting capabilities.
Prometheus Web UI: Prometheus features a web UI that visualizes all metrics by pulling them from the HTTP server endpoint. Leveraging the PromQL (Prometheus Query Language), users can pull and visualize metrics on other tools like Grafana.

OpenTelemetry Overview

Similar to Prometheus, the OpenTelemetry framework provides an alternative for collecting metrics from various services. In this diagram, the central component is the OTel Collector, responsible for scraping or collecting metrics from diverse microservices and shared infrastructure services such as Kubernetes, cloud services, and more. Once the metrics are collected, they can be persisted in a time-series database.

Notably, OpenTelemetry is observability framework-agnostic. This means that the structure of the collected data is standardized, and as OpenTelemetry is under the umbrella of CNCF(Cloud Native Computing Foundation), users have the flexibility to employ any observability framework and databases of their choice.

Incident Automation and Remediation

🔖 Definition
Involve the use of automated processes and workflows to respond to and resolve issues detected through monitoring, logging, and tracing.

🔖 Purpose
To reduce manual intervention, minimize downtime, and enhance the overall reliability of the software system.

🔖Example
There are plenty of tools for incident management but ServiceNow and PagerDuty can be always considered as the items on the top of the list.

ServiceNow Incident Automation: ServiceNow automates incident resolution through predefined workflows and scripts, reducing manual effort and accelerating response times.

ServiceNow Remediation: In ServiceNow, remediation involves corrective actions taken to resolve incidents swiftly, leveraging automated solutions and predefined processes.

Incident Automation and Remediation Real-World Example

Incident response in observability refers to the systematic approach of detecting, managing, and resolving unexpected issues or disruptions in a software system.

Let’s consider this real-world scenario

A sudden surge in website traffic leads to a spike in server CPU usage beyond normal thresholds.

1. Detection: Anomaly detection algorithms identify a sudden spike in server CPU usage, signaling a potential issue.

2. Alerting: The system triggers alerts to the operations team, notifying them of the abnormal server load and providing initial details.

3. Automated Response: Automated scripts kick in to scale up additional server instances, redistributing the load to prevent performance degradation.

4. Verification: The operations team reviews system metrics to confirm if the automated response effectively mitigated the issue, ensuring stability.

5. Notification: Upon successful resolution, notifications are sent to relevant stakeholders, updating them on the incident’s detection, response, and resolution.

Challenges and Pitfalls in Observability

Like any other aspect, Observability has its own set of challenges and pitfalls. The aspects mentioned in this picture are self-explanatory and should be clear without further elaboration.

Observability Tools

When selecting an observability tool, numerous factors come into play, influencing our choices. Considerations such as

Budget constraints
Scale of services
Data volume
Feature updates
Open-source nature
Visualization
Alerting
Keeping SDKs Up to date (coding instrumentation)

all play pivotal roles. Below, I have compiled a list of some of the top observability tools and frameworks that can be chosen based on your specific requirements.

Final Thought

To sum it up, having good observability and monitoring is important for making software successful. Getting quick insights helps fix problems fast and makes sure the application works well. Picking a platform that suits your needs is crucial because dealing with complex observability tools can be tricky. Knowing how to compare tools and find the best one for what you need is important. Understanding the basics of observability concepts can be helpful.

Follow me on Medium or YouTube if you are interested in back-end development subjects 🤞

🎬 Youtube
📖 Medium
💻 Linkedin

Cheers, Matt Ghafouri

A short review of “Building Evolutionary Architectures” book

Matt Ghafouri — Sun, 24 Sep 2023 21:36:22 +0000

I’ll summarize my takeaways from this book in a 5-minute discussion

Book Name: Building Evolutionary Architectures
Authors: Neal Ford, Rebecca Parsons, Patrick Kua
Publisher: O'Reilly Media
Publication Date: 2022 November 22 (2nd Edition)
Paperback: 262 **Pages
Language: **English
Dimensions: **7 x 0.5 x 9.25 inches
Purchase Link: Book available on Amazon

Audiences 👂🏻🔉

Software developers at all levels except for those in junior positions.

My understanding of this book 📖🔖

The author attempts to list all the important aspects of software architecture, such as scalability, ease of maintenance, security, resilience, monitoring, etc. To help choose an architecture that matches business requirements. In essence, they aim to understand what it takes to create and maintain architectures that can adapt to constant changes.

Next, the author discusses various architectural approaches, including monolithic, modular monolithic, SOA, microservice, and serverless architectures. The author evaluates these options based on fitness functions, incremental changes, and coupling to determine the most suitable architecture.

Should I Read this book?

I genuinely believe that even if you manage to extract just one page from a book, it’s worth the read. If you find the time, I highly recommend giving it a go. However, if you’re unable to do so, I hope the information you’ve gathered here serves as a valuable takeaway. Embrace it.

This article was originally published on my personal blog

Cheers, Matt Ghafouri

I’m a RICH software engineer who Creates and Sells.

Matt Ghafouri — Tue, 12 Sep 2023 08:12:28 +0000

[Article.Preface]
Success in the future will be determined by one’s ability to create and sell. We as software engineers are able to create digital products and sell them to our audience, Digital products can be an article like what you are reading now, a YouTube video, or anything else on the Internet. I will be sharing my experience about finding the solution to make passive income as a software engineer.

[Article.Authrozie]
I am a software engineer, but I’m financially illiterate,
I yearn for freedom, whether it’s financial or in terms of time.
I don’t know the secret to making money even while I sleep.
If (you can’t answer YES to at least one of these questions) then

throw a new IrrelevantArticleException(“This article is not for you”)

[Article.Reason]
I’m a software engineer, always trying to keep up with new tech and programming languages. Finance topics like the stock market and crypto trading don’t really interest me, and I don’t want to work extra on weekends on another part-time project for more money.

But, is there a way to make money with my interests? BIG Yes, definitely!

Read the full version on Medium

How does Base64 work?

Matt Ghafouri — Mon, 04 Jul 2022 07:49:26 +0000

How does #base64 works?
ASCII characters where each Base64 character contains 6 bits of binary information.
It's very useful for storing image/audio information in Strings of information. What Base64 isn't is an encryption algorithm.

There is an amazing and simple explanation for that on YouTube. check it out.

Automated dead-letter queue handling for EasyNetQ [RabbitMQ]

Matt Ghafouri — Tue, 19 Apr 2022 06:23:12 +0000

📃What is EasyNetQ?
EasyNetQ is designed to make publishing and subscribing with RabbitMQ as easy as possible. of course, you can always use the RabbitMQ client to do this, but it brings lots of complexity of maintenance cumbersome to your application.
Digging into this library is out of the scope of this article, you can read more about this incredibly simple library Here.

📃What is the dead-letter queue?

Messages from a queue can be "dead-lettered"; that is, republished to an exchange when any of the following events occur:

The message is negatively acknowledged by a consumer using basic.reject or basic.nack with requeue parameter set to false.
The message expires due to per-message TTL
The message is dropped because its queue exceeded a length limit.

🔔EasyNetQ weakness!!
Honestly. it's not a weakness, EasyNetQ tries to keep the implementation as simple as possible, because of this principle you cannot find so many features which already exist in the RabbitMQ client in this library.

But one of the biggest issues that a developer perhaps faces during the implementation of RabbitMQ functionality is dead-letter management.

EasyNetQ does not support automatic dead-letter definition. It means in case any of the conditions are mentioned, the failed message will be moved to the default error queue of RabbitMQ.

❓What's wrong with one Error queue?
Good question! dealing with different types of failed messages and monitoring the failed messages for different queues is not possible in this case. It means there should be a different dead-letter queue for each discrete queue.

🌱Solution

Here, I have implemented a Nuget package (EasyDeadLetter) for this purpose, which can be easily implemented with the minimum changes in any project.

All you need to do is follow the four steps :
1- First of all, Decorate your class object with QeueuAttribute

2- The second step is to define your dead-letter queue with the same QueueAttribute and also inherit the dead-letter object from the Main object class.

3- Now, it's time to decorate your main queue object with the EasyDeadLetter attribute and set the type of dead-letter queue.

4- In the final step, you need to register EasyDeadLetterStrategy as the default error handler (IConsumerErrorStrategy).

That's all. from now on any failed message will be moved to the related dead-letter queue.

📑Resources
NuGet Package

MattGhafouri / EasyDeadLetter

RabbitMQ automatic DeadLetter handler for dotnet | EasyNetQ

🌱Easy Dead-Letter 🌱

📕Handling dead-letter queues in EasyNetQ (RabbitMQ)

Messages from a queue can be "dead-lettered"; that is, republished to an exchange when any of the following events occur:

The message is negatively acknowledged by a consumer using basic.reject or basic.nack with requeue parameter set to false.

The message expires due to per-message TTL;

The message is dropped because its queue exceeded a length limit

🔔 Problem

If you are using RabbitMQ client for dotnet there is no problem, you can define dead-letter queues during a queue declaration, But using RabbitMQ client brings lots of implementation complexity to the application, because of that so many developers prefer to use the EasyNetQ library which is a wrapper on RabbitMQ client(Amazing and simple) There is no implementation for dead-letter queues in EasyNetQ to keep the library easy to use, It means in case of any exception in the message consumers,…

View on GitHub

🤞If you want to support me, please share this article with your community.

Follow me on Linkedin

Control concurrency for shared resources in distributed systems with DLM (Distributed Lock Manager)

Matt Ghafouri — Fri, 24 Sep 2021 15:45:03 +0000

Scenario
Imagine in our application we need to update a value that is shared between all requests. It means we should control concurrent requests for manipulating this shared resource. Somehow we should serialize the access to resources. In our example, we keep this value in the Redis cache for reducing latency.
To make our example more clear, imagine we have a contribution amount that is updated by each request. This value should be plus the contribution amount of incoming requests.
Let’s demonstrate our scenario with graphic symbols for better understanding.

We have an application that has subscribed to a RabbitMQ queue. In the simplest model, we receive a message from the queue and update our contribution amount in Redis in synchronized mode, which means the application process requests one by one. Everything looks fine.

Problem

We may encounter two problems :
1- What if the application uses async processing to process messages from the queue?

2- What if we horizontally scale the application, means running several instances of the application?

In these two cases, we will receive more than one request, and somehow the concurrency should be controlled, otherwise, we will face Race conditions in Redis and the contribution value is not valid anymore.

The first solution which can be brought up is to use an internal lock object for controlling the critical point.
This solution works well for the first problem. But what about the second one?

In the case of horizontal scaling, the internal lock just works in the scope of one instance, we should find a control mechanism outside of the application scope.

Solutions
1- Actor systems like Akka or Orleans(.Net)
2- Distributed lock manager (DLM)
3- Add your solutions here

In my point of view if your application does not have to deal with lots of concurrent scenarios like what I’ve mentioned, using a DLM is better for sake of simplicity. Because implementing actor systems bring more complexity to your projects.

But if you have an application with lots of critical points in which concurrency should be controlled, I also prefer to use actor systems. The point is, in the example which I’ve mentioned, we just need to control access for a single shared value. So I prefer to use DLM instead of bringing extra complexity to my application code.

Ok, forget the Actor systems, and Let’s talk about DLM and how it can be helpful for our problems 😊
For implementing a DLM we have several choices. for example, we would use Zookeeper or Redis or maybe other tools. As I have experience with Redis DLM, so I’ve decided to explain it in this article.

I don’t want to talk about Redis DLM in-depth, because it’s already been explained well on Redis Official website.

I’ve just tried to define a scenario and solve it with the help of Redis DLM.

A Redis Enterprise cluster node can include between zero and a few hundred Redis databases in one of the following types:

· A simple database, i.e. a single master shard
· A highly available (HA) database, i.e. a pair of master and slave shards
· A clustered database, which contains multiple master shards, each managing a subset of the dataset (or in Redis terms, a different range of “hash-slots”)
· An HA clustered database, i.e. multiple pairs of master/slave shards

I made a contact with the Redis development team and I asked about Redis DLM best practice. As they said, using clustered Redis with more than 2 Master nodes. Although you can use the RedLock algorithm with One Master and several slaves ( leader-follower model).

Before starting to Implement a Redis DLM, it’s better to notice these points:

1- DLM in Redis uses the RedLock algorithm to controlling lock management.

2- We have several implementations of the RedLock algorithm in different languages Redis Official website. and you do not need to implement it again.

3- Our lock manager also should be configured with expiration time for locks, because without expiry time you may face deadlock, to avoid this it’s better to set expiry time. It means after that time the lock will be released automatically by Redis.

4- We also should create a mechanism as a retry pattern to retrying for acquiring the lock. If the requests couldn’t get the lock, we should consider it as a failed request and it should be controlled ( for example we can republish the failed request to the queue, to receive it again or we could use an outbox pattern to process them in a background worker.)

5- I highly recommend you to find an implementation of RedLock in your using language which supports retry pattern and also expiry time for the lock.

So let’s define our scenario step by step with the help of Redis DLM.

1- We have an application that has subscribed to a queue.

2- We use async processing and also we want to horizontally scale our application which means we will have several handlers for a specific queue, and we will receive several messages simultaneously.

3- All the requests want to update a contribution value which is located inside the Redis cache. It means our shared resource is the contribution value and concurrency should be controlled for updating this value in Redis.

4- We use Redis distributed lock manager (DLM) as a coordinator to serialize the access to this contribution amount.

5- each request which wants to update the contribution amount should acquire a lock from DLM, if it acquired the lock it could be able to update the contribution amount in Redis.

Conclusion

In modern applications in which async processing and horizontal scaling can be considered as a Must-have, we will face concurrent requests, and If we have some shared resource, somehow this concurrency must be controlled.

For controlling concurrency in applications we could implement different approaches, but you should always consider choosing the best approach because a good developer avoids bringing unnecessary complexity to an application by over-designing or overthinking.

So, for concurrency management in an application, first, you must understand the problem, after that determine your critical points, and in the last step according to the collected data and consulting with a domain expert, you can choose your approach. Distributed lock managers in most cases can handle your problems. But sometimes you have to choose another approach like Actor systems.

That’s all, Hope you’ve enjoyed the article, feel free to contact me and send me your comments.

This article was originally published on my personal blog

Cheers, Matt Ghafouri

Microservice Roadmap

Matt Ghafouri — Tue, 31 Aug 2021 20:00:02 +0000

Why Microservice architecture?

Nowadays with the rise of social media, fast internet, etc. The tendency to use applications is getting more and more. As a result of these behavior changes, monolithic applications need to deal with a tremendous majority of changes.
most of the businesses face a new feature that should be added to their application, on the other side, the amount of data that should be processed is getting much higher and in so many cases monolithic applications because of their drawbacks such as slow development velocity, slower deployment, as barely support the agile methodology which is the main idea to deal with fast changes in the software industry.
If you want to create a software project for a big or complex business (It’s not true about startup projects or those projects in the funding stage, because they have to be submissive to the time to market limitations) it’s better to start with microservice architecture.
So, one of the solutions to boost the application flexibility, scalability, and so many other concerns is to follow a flexible architecture like microservice-based architecture.

Why do you need a roadmap?

As I understand so many developers, want to know how they should start this journey, obviously, there are thousands of resources that can be used, but the problem is the dispersion of resources. I decided to make this journey more clear by defining a road map for this learning curve.

How does this roadmap work?

A microservice-based architecture has several discrete units which they work together to receive and process requests from various sources. Some parts of this complex can be plug-in units, which means in case of need either you can plug a new unit or unplug a unit, which shouldn’t disturb the overall work of an application.

For example, if you have decided to implement a microservice architecture, you should be familiar with various concerns in a life cycle of an application such as persistence, logging, monitoring, load balancing, caching, etc. besides you should know which tools or which stack are more suitable in your applications.

Well, this is the main approach of my article, I will explain the concern, then introduce some of the best tools for satisfying the needs.

I explain each topic to get answer these questions:

What is it?
Why should I use it?
Which tools are better?

Notice that I have mentioned for tools part just two or three tools, Of course, we have so many other tools, the criterion for choosing these tools is popularity, performance, being open-source, and also update frequency.
Notice again we have also cloud-based services which are out of the scope of this article to be discussed.

I have used the above diagram as the architecture diagram sample, because this diagram involves most of the microservice architecture components and also has been recognized as a standard model, of course, I know this diagram is not a very comprehensive diagram, but I have tried to keep this article as simple as possible, maybe in future I would revise the article and write a new version of that. I think the best mindset about explaining every concept, should always follow the KISS pattern.
The concepts which have been explained in this article are:

• Docker
• Container orchestration
• Docker container management
• API gateway
• Load Balancing
• Service discovery
• Event Bus
• Logging
• Monitoring And Alerting
• Distributed tracing
• Data Persistence
• Caching
• Cloud Provider

                            Docker

What

Docker is an open-source platform for containerizing your applications, with the libraries, and dependencies that your application needs to run in various environments. with the help of Docker, development teams are able to package applications into containers.
Why
Actually, Docker is one of the tools to containerize applications, Which means you are also able to create containers without using Docker, The real benefits of Docker is to make this process easier, safer, and much simpler.
Tools
Docker Inc.

                   Container Orchestration

What
After you containerize your application, you will need some tools for managing containerized applications for doing some manual and automated operations like horizontal scaling.
Why
These tools provide some services for your application management like automated load-balancing, guarantee high service availability.
This kind of service is done by defining several manager nodes, and in case of any failure in one node manager, the other managers can keep the application services available.
Tools
Kubernetes or K8s ,Docker Swarm

                   Docker Container Management

What
Managing Docker environment, configuration management, and providing the environment security, etc. These concerns can be centralized and automated by a docker container management tool.
Why
A Docker container management tool provides a GUI-based tool for users, and they do not have to deal with CLI which is not comfortable for all users. These tools give developers a rich UI to build and publish their images, also make it much easier to do some operational tasks such as services horizontally scaling by providing a simplified user interface.
Tools
Portainer, DockStation, Kitematic, Rancher

                           API Gateway

What
An API gateway can be considered as an API management tool that works as a middleware between your application services and different clients. An API gateway can manage many things such as:

Routing :a gateway receives all API requests and forwards them to the destination services.

Logging : you will be able to logs all the requests in one place.

Authorization: check if you are as a user are eligible to access the service or not, if not the request can be short-circuited

Performance profiling: you can estimate every request execution time and check your application bottlenecks.

Caching: with handling your caching at the gateway level, you would eliminate so much traffic on your services.

In fact, It works as a reverse proxy, and clients just need to know about your gateway, and application services can be hidden from the outside.
Why
Without an API gateway, you may need to do some cross-cutting concerns in every service, for example, if you want to log the request and response for services.
Besides, if your application consists of several services, your client needs to know about each service address, and in case of changing a service address, several places should be updated.
Tools
Kong, Ocelot

                      Load Balancing

What
One of the most important reasons that we have chosen the microservice architecture is scalability, Which means we will be able to handle more requests by running more instances of our services, but the question is, which instance should receive requests or How do clients know which instance of service should handle the requests?
The answer to these questions is Load Balancing. Load Balancing means sharing the income traffic between a service instance.
Why
In order to scale your independent services, you need to run several instances of the services. With a load balancer, clients do not need to know the right instances which serving.
Tools
Traefik, NGINX, Seesaw

                        Service Discovery

What
As your application services count getting more and more, services need to know each other service instance addresses, but in large applications with lots of services, this cannot be handled. So we need service discovery which is responsible to provide all components addresses in your application, they could easily send a request to the service discovery service and get the available services instance address.
Why
When you can have several services in your application, service discovery is a must-have for your application.
Your application services' don't need to know each service instance address, it means service discovery paves this way for you.
Tools
Consul, Zookeeper, Eureka, etcd and Keepalived

                          Event Bus

What
In microservice architecture pattern, you would use two different types of communication, Sync and Async communication
Sync communication means services call each other through HTTP call or GRPC call.
Async communication means services interact with each other via a message bus or an event bus, It means there is no direct connection between services.
While your architecture can use both communication styles simultaneously, for instance in an online shop example you can send a message whenever an order is registered and those services which have subscribed to a specific channel will receive the message. But sometimes you may need some real-time inquiry, for instance, you need to know the quantity of an item, you may use GRPC or HTTP call between services to get the response.
Why
If you want to have a scalable application with several services, one of the principles you would follow is to create loosely coupled services that interact with each other through an event bus. Also, if you need to create an application that is able to plug in a new service to receive a series of specific messages, you need to use an event bus
Tools
RabbitMQ, Kafka

                         Logging

What
When using a microservice architecture pattern it’s better to centralize your services logs. These logs would be used in debugging a problem or aggregating logs according to their types for analytical usages.
Why
In case of any need to debug a request, you may face difficulty if you do not gather services logs in one place. You are also able to correlate the logs related to a specific request with a unique correlation Id. It means all logs in different services related to a request will be accessible by this correlation Id.
Tools
Elastic Logstash

                    Monitoring And Alerting

What
In a microservice architecture, if you want to have a reliable application or service you have to monitor the functionality, performance, communication, and any other aspect of your application in order to achieve a responsible application.
Why
You need to monitor the overall functionality and services health, also need to monitor performance bottlenecks and prepare a plan to fix them. Optimize user experience by decreasing downtime of your service by defining early alerts for your services in critical points. Monitor overall resource consumption of services, when there are under heavy load and etc.
Tools
Prometheus, Kibana, Grafana

                    Distributed Tracing

What
Debugging is always one of the developers' concerns, As you, all have the experience to trace or debug a monolithic. It's very straightforward and easy, But when it comes to microservice architecture, because a request may be passed through different services, which makes it difficult to debug and trace because the codebase is not in one place, so here distributed tracing tool can be helpful.
Why
Without a distributed tracing tool it's frustrating or maybe impossible to trace your request through different services. With these tools, you can easily trace requests and events with the help of having a rich UI for demonstrating the flow of the request.
Tools
OpenTelemetry, Jeager, Zipkin

                     Data Persistence

What
In most systems, we need to persist data, because we would need the data for further processes or reporting, etc in order to persist data, we need some tool to write our applications data into physical file with different structure.
Why
In monolithic applications we used to have one or two different persistence types and most of monolithic applications use Relational databases like SQL Server, Oracel, MySQL. But in microservice architecture we should follow “DataBase per service” pattern, it means keep each microservice’s persistent data private to that service and accessible only via its API.
Well,You would have different databases for different usages and scenarios.For instance a report service might use NoSQL databases like ElasticSearch or MongoDB, because they use document base structure, It means the structure for persisting data in these databases is different from relational databases, which are suitable for services with high rate of reading or fetching data.
In the other hand you may need relational databases like Oracle or SQL SERVER in some microservices or you may also need some databases which support graph structure or key-value structure.
So, In Microservice architecture according to the service mission you would need different types of databases.
Tools
Relational databases or RDBMS : PostgreSQL, MySQL, SQL SERVRE, Oracle
NoSQL database : MongoDB, Cassandra, Elasticsearch

                      Caching

What
Caching reduce latency in service-to-service communication of microservice architectures. A cache is a high-speed data storage layer that stores a subset of data. When data is requested from a cache, it is delivered faster than if you accessed the data’s primary storage location.
Why
In a microservice architecture there are many strategies which caching can be implemented in those ways. Such as

1- Embedded Cache (distributed and non-distributed)
2- Client-Server Cache (distributed)
3- Reverse Proxy Cache (Sidecar)

In order to reduce latecny, caching can be implemented in different layers. It means you would have a higher response time. besides you can also implement distributed caching which is accessable for several microservices and sometimes considered as a placeholder for shared resources, which have different usage, like
Rate-limiting : The most common reason for rate limiting is to improve the availability of API-based services by avoiding resource starvation.
Tools
Redis (REmote DIctionary Server), Apache Ignite, Hazelcast IMDG

                     Cloud Provider

What
A cloud service provider is a third-party company offering a cloud-based platform, infrastructure, application, or storage services. Much like a homeowner would pay for a utility such as electricity or gas, companies typically have to pay only for the amount of cloud services they use, as business demands require.Microsoft

Most important categories for cloud providers
1- Software as a Service (SaaS).
2-Platform as a Service (PaaS).
3- Infrastructure as a Service (IaaS).

Why
One benefit of using cloud computing services is that firms can avoid the upfront cost and complexity of owning and maintaining their own IT infrastructure, and instead simply pay for what they use, when they use it. Today, rather than owning their own computing infrastructure or data centers, companies can rent access to anything from applications to storage.
Tools
Amazon Web Services (AWS) , Microsoft Azure, Google Cloud, Alibaba Cloud

Conclusion

In this article, I tried to demonstrate a roadmap which associated with microservice architecture pattern. If you want to implement a microservice architecture from scratch or migrate your monolithic to microservice architecture, you will need to know these concepts.
Besides these concepts, we have other concepts like service mesh, caching, persistence that could be a part of this roadmap, but I intentionally have not mentioned them, for sake of simplicity and with a view to brevity.
Feel free to send me your comments and suggest other concepts to this roadmap.

Follow me on Medium

Writing logs into Elastic with NLog , ELK and .Net 5.0 (Part 2)

Matt Ghafouri — Thu, 12 Aug 2021 18:38:50 +0000

In the previous article I demonstrated how we can write our log in our dotnet applications into Elastic with NLog, if you haven't read that article, I highly recommend reading it first.

After I published the first part, I received some comments that said, when they try to write logs into Elastic they have faced some problems.

I checked some of the cases and realized, they have enabled Basic authentication in Elastic. So, whenever NLog tries to write logs into Elastic by calling the endpoint of the Elastic, they get an authentication error.


OriginalException: Elasticsearch.Net.ElasticsearchClientException: Failed to ping the specified node.. Call: Status code 401 from: HEAD /
---> Elasticsearch.Net.PipelineException: Failed to ping the specified node.
---> Elasticsearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.
at Elasticsearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IElasticsearchResponse response)
at Elasticsearch.Net.RequestPipeline.Ping(Node node)
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.Ping(Node node)
at Elasticsearch.Net.Transport`1.Ping(IRequestPipeline pipeline, Node node)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
# Audit exception in step 1 PingFailure:
Elasticsearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration.
at Elasticsearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IElasticsearchResponse response)
at Elasticsearch.Net.RequestPipeline.Ping(Node node).

To solve the problem you should set credentials in your nlog.config file. so change the nlog.config file like this.

requireAuth="true"
username="*******"
password="********"

After I added these attributes to my target everything was working well. But the other issue reported was the SSL connection problem. If your Elastic endpoint has SSL, you may also face this error.


Elasticsearch.Net.ElasticsearchClientException: Failed to ping the specified node.. Call: Status code unknown from: HEAD /
---> Elasticsearch.Net.PipelineException: Failed to ping the specified node.
---> Elasticsearch.Net.PipelineException: An error occurred trying to write the request data to the specified node.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: PartialChain
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken).

One solution to fix this issue is disabling certificate validation on Elastic endpoint. To achieve this, you need to add another attribute to your Elastic target.

DisableCertificateValidation="true"

Now your Elastic target in nlog.config file should be something like this:


<target xsi:type="ElasticSearch" 
        name="elastic"               
        index="MyService-${date:format=yyyy.MM.dd}" 
        layout ="MyService-|${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" 
        includeAllProperties="true"
        requireAuth="true" 
        username="*******"
        password="*********" 
        uri="https://elasticSampleaddress.com:9200" />

That all. I hope you find this article is useful, let me know if you have any comment.

Read Writing logs into Elastic with NLog , ELK and .Net 5.0 (Part 1)

Follow me on Medium

Projects Folder Structures Best Practices

Matt Ghafouri — Sat, 07 Aug 2021 16:13:45 +0000

Structuring a solution’s folder is one of the first steps toward starting any software project. If you want to have a good structure with a good separation of tasks, you should be familiar with best practices.
For this purpose, one of the good references which can be used is Github's good projects’ repositories.

Why does folder structure matter?

1- It makes it easier to understand the structure of the project, especially for new developers when they join a new team.

2- It helps to separate deployment from application codes ( test codes vs application codes)

3- defining different categories for different objects(DTOs, Contracts, Filters, etc)

4- defining categories for various assets which can be set access permission or applying some other concerns on them for better management.

Well, so far we understand the importance of the folder structure of software project solutions.
Important to keep in mind, in a microservice architecture, we may have different folder structures because according to the size of the project or other concerns (framework, language, etc), we may see different structures. but in most cases, you could use this structure for your projects.

Let’s check an example
Here We are going to define a folder structure from scratch. In this example, I am going to use Microsoft Visual Studio as IDE, but the concept is the same if you want to use different IDE or languages.

First of all, create a repository on Github with the showed properties

As you create your repository, the next step is to clone the project on your local system. Open Visual Studio, select Clone repository from the File menu in the opened window, paste the Git URI and click on the Clone button.

you will see the existing items in the solution explorer panel

As you can see in the below image, select Project… from the File menu, then select ASP.NET Core Empty project type.(obviously, you can select your own project type)

In the Configuration window, set the properties and click on the Next button

Now, I want to use a Git command to create a .sln file for our solution, if you haven’t installed GitBash on your local system, use this link to install.

Git is a set of command line utility programs that are designed to execute on a
Unix style command-line environment. ReadMore

Open the cloned folder in the FileExplorer.

As the next step, in the opened Window highlight the address bar and type CMD, then hit enter; The Command Prompt will be shown, type the command in order to create a .sln file

dotnet new sln

To open with the Visual Studio, double click on the newly created solution file. You would have a solution with this structure if you have followed the previous steps.

Open the root folder and create two folders besides the solution file (src and test)

From now on every project which related to this solution should locate inside the src folder and all the test project should locate inside the test folder.

Here you can add all other projects according to your needs (If you use Visual Studio as IDE, take into consideration that the creation of a folder in Visual Studio won't create a physical folder, and you need to create it manually.)

You also may need to add other folders besides your project like:
docs, tools, releases

Conclusion
Structuring folders in a solution is one of the most important steps in software project implementation.
A good beginning makes a good ending

Follow me on Medium

Maybe you like :
Writing logs into Elastic with NLog , ELK and .Net 5.0

Writing logs into Elastic with NLog , ELK and .Net 5.0 (part 1)

Matt Ghafouri — Wed, 04 Aug 2021 10:25:58 +0000

If you are using Microservice-based architecture, one of the challenges is to integrate and monitor application logs from different services and ability to search on this data based on message string or sources, etc. You can also read microservice roadmap here.

This post assumes you already have a general idea of Elastic and Dotnet core and what it's used for, so if it's new to you, take a look at the Elastic official website.

So, what is the ELK Stack?
“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

On the other hand, NLog is a flexible and free logging platform for various .NET platforms, including .NET standard. NLog makes it easy to write to several targets. (database, file, console) and change the logging configuration on-the-fly.

If you combine these two powerful tools, you can get a good and easy way for processing and persisting your application’s logs
In this article, I’m going to use NLog to write my .Net 5.0 web application logs into Elastic. after that, I will show you how you can monitor and search your logs with different filters by Kibana.
First of all Open Visual Studio and select a new project (ASP.NET Core Web API)(If you want to know the best practice for Folder structure in your project, read my article)

On the next page set the target framework on .Net 5.0(current) and also check Enable OpenAPI support option for using swagger in the project.

We need some Nuget packages, In order to install them, from the Tools menu select Nuget Package Manager and then select Manage Nuget packages for solutions.
In the opened panel search the below packages and install them one by one

NLog.Web.AspNetCore
NLog.Targets.ElasticSearch

After you installed the packages, you need to add a config file for NLog. In order to add the file, right-click on the current project in the solution and select add => new Item, then add a web configuration file and name it nlog.config

Open the newly added file and paste the below codes



<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">

    <extensions>
        <add assembly="NLog.Web.AspNetCore"/>
        <add assembly="NLog.Targets.ElasticSearch"/>
    </extensions>


    <targets async="true">

        <target name="elastic" xsi:type="ElasticSearch" index=" MyServiceName-${date:format=yyyy.MM.dd}"
                uri="http://localhost:9200"
                layout ="API:MyServiceName|${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" >
        </target>

    </targets>

    <rules>
        <logger name="*" minlevel="Debug" writeTo="elastic" />
    </rules>
</nlog>

In the target tag, we define our configuration like Elastic service URI, layout pattern, Elastic index name and etc.
Let’s take a look at these tags and their attributes

Index: MyServiceName -${date:format=yyyy.MM.dd}: it means we will have different index for every day’s log. For example, MyServiceName-2021.08.03 related to all the logs of the third of August

ElasticUri: Elastic URI address.

Layout: your log message pattern, you can determine which information you want to have in the message of your log.

In the rules tag, the minimum level for the writing log is defined.

You can read more detail about nlog tag config Here

After we understand the structure and meaning of the config file, the next step is to set up NLog logger instead of .Net build-in logger. To achieve this, Open program.cs and change Main method like this



var logger = NLogBuilder.ConfigureNLog("nlog.config").GetCurrentClassLogger();
            try
            {
                logger.Debug("init main");
                CreateHostBuilder(args).Build().Run();
            }
            catch (Exception exception)
            {
                logger.Error(exception, "Stopped program because of exception");
                throw;
            }
            finally
            {
                NLog.LogManager.Shutdown();
            }

Change the CreateHostBuilder in program.cs as well:



public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                })
                .ConfigureLogging(logging =>
                {
                    logging.ClearProviders();
                    logging.SetMinimumLevel(LogLevel.Trace);
                })
                .UseNLog();  // NLog: Setup NLog for Dependency injection

In the next step, open Startup.cs and in the ConfigureService method add below code



services.AddLogging();

Now you are able to write your log into elastic with Injecting ILogger in every class constructor.



[ApiController]
    [Route("[controller]")]
    public class WeatherForecastController : ControllerBase
    {
        private ILogger<WeatherForecastController> _logger;

        public WeatherForecastController(ILogger<WeatherForecastController> logger)
        {
            _logger = logger;
        }

        [HttpGet]
        public IActionResult Get()
        {   
            _logger.LogDebug("Debug message");
            _logger.LogTrace("Trace message");
            _logger.LogError("Error message");
            _logger.LogWarning("Warning message");
            _logger.LogCritical("Critical message");
            _logger.LogInformation("Information message");

            return Ok();
        }
    }

From now on, If you run your application and send a request with Postman, all types of logs will be written into Elastic.

Well, We successfully configured our application to write the logs into Elastic, but how we can monitor and search out logs. For this purpose, Elastic has Kibana as a tool that lets users visualize data with charts and graphs in Elasticsearch.
After you install the Elastic, you can have access to Kibana

First of all, we should be able to create a pattern which our application's log will be fetched by that pattern. For doing this, select the Management option in the left panel

In the management panel select Index Pattern link in the left side

In the showed panel click on Create index pattern button. You will see Create index pattern panel, there is an Index pattern input box which you should input your Index name(in our example we set the index name in the nlog.config file like this MyServiceName-${date:format=yyyy.MM.dd}.)
If you want your pattern to include wild character instead of the date you should write down myservicename-* in the input(it means we want to create a pattern that is able to load all logs with a prefix of myservicename- ).

If there were any logs with this pattern, a Success message will be shown beneath the input box. You can select the pattern and click on Next step.

In the next panel, you can add a Time filter to your pattern, which means you will be able to filter logs according to the insertion date. After you selected @timestamp, click on Create index pattern button to complete this process.

Now click on the discover link on the left panel to redirect to the logged dashboard.

As you see, we have several logs with different types related to our application. If you do not see any log, be sure that you have selected your newly defined pattern on the left side of the dashboard

For filtering our logs according to the log message or insertion date you can use the filter bar located beneath the menu bar.

For example, in this picture, I have search logs that contain critical word in their log messages.
After you set your filter click on the green update button on the right side.

That’s it. Now you can easily filter your application's logs with different patterns.

For more information about Kibana features you can use this link

Conclusion
Aggregating application logs is one of the benefits of Elastic logstash and NLog, but with this approach, you also can easily export your application logs in any format and also could create aggregate logs which is very helpful for analyzing your logs according to the type of logs.

Read Writing logs into Elastic with NLog , ELK and .Net 5.0 (Part 1)
Follow me on Medium

Maybe you like :
Projects Folder Structures Best Practices