DEV Community: Ajit Chelat

How AIOps Helps in Application Monitoring

Ajit Chelat — Wed, 07 Jul 2021 16:10:53 +0000

There’s no one-size-fits-all approach regarding application monitoring, especially for companies using applications in various cloud environments. Companies are rapidly investing in microservices, mobile apps, data science programs, data ops, etc. Subsequently, they’re also integrating monitoring tools to improve domain-centric monitoring abilities.

AIOps tools help streamline the use of monitoring applications. It allows companies that need high application services to efficiently manage the complexities of IT workflows and monitoring tools. AIOps extends machine learning and automation abilities to IT operations. These robust technologies aim to detect vulnerabilities and issues to resolve them, determine operational trends, and simplify the remediation of the problems that affect their applications’ performance and availability.

What Exactly Is AIOps?

AIOps is short for Artificial Intelligence for IT Operations. AIOps combines machine learning, data analytics, and many other AI technologies to automate the identification and remediation of common and recurring IT operations issues. AIOps leverages data from logs and event recordings to monitor assets and obtain visibility into dependencies without interfering with IT systems.

Capabilities of AIOps Platforms

AIOps platforms provide the following capabilities:

Machine learning capabilities to help in identifying patterns in the collected data.
A dedicated data platform for aggregating raw data and logs from various monitoring tools and data sources across your applications and infrastructure.
Dashboards, analytics, and console integration help IT operations gain a single-pane view over their applications and infrastructure.
Out-of-the-box integrations with tools used for IT service management, monitoring, agile development, collaboration, and log data collection, parsing, and ingestion tools.

How Does AIOps Work?

AIOps platforms are powered by algorithms that automate and simplify prominent aspects of IT operations and application monitoring:

Data Selection: It collects all the data generated by applications and infrastructure in the form of logs and events and analyzes it. Post analysis, AIOps platforms highlight data that has an issue.
Pattern Discovery: AIOps platforms correlate and find relationships between different data elements in the form of patterns.
Interference: AIOps determines the root causes of new and recurring issues allowing companies to take proactive actions to mitigate the implications of these issues.
Collaboration: AIOps platforms simplify and promote collaboration across IT teams through unified dashboards and intelligent notification systems.
Automation: AIOps works towards automating responses to issues and threats as much as possible, thereby making issue and threat remediation quick and straightforward.

Improved Application Monitoring with AIOps

The adoption of AIOps has numerous benefits – right from processing data from multiple sources faster and using that data to make data-driven decisions, to making IT operations more proactive by predicting and remediating performance issues across applications and deployments. Let’s take a closer look at how AIOps is helpful in improving your application monitoring efforts.

Detect Hidden Relationships

IT operations and monitoring are an extensive web of interdependencies; no system works independently. However, with so much data present, it is challenging to understand the relationships between systems. AIOps allows you to evaluate performance metrics across different types of systems quickly. This can help identify the impact of IT applications on the overall company’s performance and customer satisfaction.

This is accomplished by initially working with the business to determine mission-critical activities for such applications. The next step is to gather data produced during the day-to-day tasks like orders, cancellations, transactions, etc. AIOps algorithms can be leveraged to identify patterns or clusters in the collected data, allowing businesses to understand the relationships better.

Optimizing The Use of Customer And Transaction Data

Capabilities of AIOps can help in the identification of patterns, anomaly recognition, categorization, and extrapolation. These are essential aspects of big data analytics operations that the organization applies to the transaction and customer data. Leveraging AIOps can help in understanding user behavior in broad IT systems.

This will make it easier to monitor how any modifications on the applications will affect the business operations. By harnessing internal application monitoring data, AIOps can bring together customer and transaction data effectively. When the information is readily available, a business can efficiently choose the right path for the application.

Forecasting The Issues

An essential role of AIOps is ameliorating the predictive analytics activities. It closely studies the current and past behavior of the apps. This allows the technology to predict future scenarios, enabling the business to adjust its strategies. This proactive approach helps in improving application performance and also in gaining competitive advantages.

For instance, companies can identify changing trends in how users are interacting with apps. So they will have a clear idea of the areas that they need to focus. Moreover, AIOps allows businesses to perform a deep analysis of the cause of the problem. Not just that, it will also take the necessary steps to eliminate the issue before it impacts the performance.

Decrease The Response Time

By leveraging AIOps, companies can reduce the response time of dealing with errors and outages. Experts believe that AIOps can reduce the cost of events like errors, outages by 30% to 40%. This signifies a massive saving considering that the average cost that a company bears in service disruption is approximately $300,000 per hour.

This is due to the ability of this powerful technology to detect where the data originates. Every system that a business uses produced a lot of data, making it harder to track the source of information. But AIOps manages the massive amount of data from a central location, allowing better process and application security.

Bringing Together Silos

One of the hurdles in improving application performance is how siloed organizations can be. More than 90% of IT professionals say that most monitoring tools only provide them with information related to their areas of responsibility.

But AIOps can deal with this issue by leveraging data analytics and machine learning. These technologies allow the tools to monitor tons of information streams. Such extensive monitoring makes it easier to spot problems that would otherwise be difficult to spot with a siloed approach.

Final Thoughts

IT leverages a lot of application monitoring tools to maintain operational efficiency. However, each of these tools collects a massive amount of data that needs to be maintained. The team fails to detect vulnerability and issues in the complex web of data, leading to security threats. By harnessing the potentials of AIOps, IT teams can automate and improve their application monitoring processes by leaps and bounds

How to stream AWS CloudWatch logs to LOGIQ

Ajit Chelat — Sat, 26 Jun 2021 10:51:06 +0000

AWS CloudWatch is an observability and monitoring service that provides you with actionable insights to monitor your applications, stay on top of performance changes, and optimize resource utilization while providing a centralized view of operational health. AWS CloudWatch collects operational data of your AWS resources, applications, and services running on AWS and on-prem servers in the form of logs, metrics, and events. CloudWatch then uses this data to help detect and troubleshoot issues and errors in your environments, visualize logs and metrics, set up and take automated actions, and uncover insights that help keep your applications and deployments running smoothly.

AWS CloudWatch provides excellent observability for your applications and infrastructure hosted on AWS. But what about your applications and resources hosted on service providers? While you can stream their logs into CloudWatch using proxies and exporters, it isn’t that straightforward. You’d have to monitor them separately using a your service provider’s own monitoring tool or build something in-house using Prometheus or Grafana, maybe. Why train your eyes to watch multiple monitoring tools when you can centralize monitoring and observability across your on-premise servers and cloud providers with LOGIQ? LOGIQ plugs into numerous data sources to centralize your logs and visualize them in a single pane regardless of the service provider.

You can easily stream your AWS CloudWatch logs into LOGIQ, thereby letting you monitor your AWS resources applications along with everything else you’re watching with LOGIQ. You can also visualize and analyze your AWS CloudWatch logs in real-time and gain powerful insights into their performance and security.

This guide will show you how you can stream your AWS CloudWatch logs into LOGIQ in no time. You can get yourself a free-forever instance of the LOGIQ PaaS Community Edition and try out the steps listed in this article to stream your AWS CloudWatch logs to LOGIQ.

LOGIQ’s AWS CloudWatch Exporter Lambda function

Since we love keeping it simple at LOGIQ, we’ve built an AWS Lambda function that enables you to export your CloudWatch logs to your LOGIQ instance. This AWS Lambda function acts as a trigger for a CloudWatch log stream.

Creating the LOGIQ CloudWatch Exporter Lambda Function

You can create the LOGIQ CloudWatch Exporter Lambda Function using the CloudFormation template available at https://logiqcf.s3.amazonaws.com/cloudwatch-exporter/cf.yaml.

Note: Alternatively, you can also use the code available in our client integrations Bitbucket repository to create the Lambda function.

This CloudFormation template creates a Lambda function along with the permissions it needs. Before using this template, you’ll need to configure the following attributes:

Parameter	Description
`APPNAME`	A readable application name for LOGIQ to partition logs by.
`CLUSTERID`	A Cluster ID for LOGIQ to partition logs by.
`NAMESPACE`	A namespace for LOGIQ to partition logs by.
`LOGIQHOST`	IP address or hostname of your LOGIQ instance.
`INGESTTOKEN`	JWT token to securely ingest logs into LOGIQ

Creating and configuring the CloudWatch trigger

Once you’ve created the AWS Lambda function, it’s time to create and configure the CloudWatch trigger. On your AWS dashboard, do the following:

Navigate to the AWS Lambda function you just created (logiq-cloudwatch-exporter).
Click Add Trigger.
On the Add Trigger page, select CloudWatch Logs.
Next, select the Log group you’d like to stream to LOGIQ.
Enter a Filter name and optionally add a Filter pattern.

And that’s it! All new logs from the CloudWatch log group you configured are streamed directly to your LOGIQ instance.

From here, you can easily view, query, visualise and analyse your CloudWatch logs while detecting anomalies in real-time thereby helping you keep your AWS applications and resources always on and performing at their best.

If you enjoyed trying out this guide and the Community Edition of LOGIQ PaaS, let us know in the comments. You can also reach out to us if you'd like a detailed demo of the LOGIQ Observability platform and witness first-hand how LOGIQ can help you derive more value from your log data.

Shipping and Visualizing Jenkins Logs with LOGIQ

Ajit Chelat — Fri, 25 Jun 2021 14:54:23 +0000

Jenkins is by far the leading open-source automation platform. A majority of developers turn to Jenkins to automate processes in their development, test, and deployment pipelines. Jenkins’ support for plugins helps automate nearly every task and set up robust continuous integration and continuous delivery pipelines.

Jenkins provides logs for every Job it executes. These logs offer detailed records related to a Job, such as a build name and number, time for completion, build status, and other information that help analyze the results of running the Job. A typical large-scale implementation of Jenkins in a multi-node environment with multiple pipelines generates tons of logs, making it challenging to identify errors and analyze their root cause(s) whenever there’s a failure. Setting up centralized observability for your Jenkins setup can help overcome these challenges by providing a single pane to log, visualize, and analyze your Jenkins logs. A robust observability platform enables you to debug pipeline failures, optimize resource allocation, and identify bottlenecks in your pipeline that hamper faster delivery.

We’ve all come across numerous articles that discuss using the popular ELK stack to track and analyze Jenkins logs. While the ELK stack is a popular service for logging and monitoring, its use can be a little challenging. While the ELK stack performs brilliantly in simple, single-use scenarios, it struggles with manageability and scalability in large-scale deployments. Additionally, their associated costs (and changes in Elastic Licensing) might raise a few eyebrows. LOGIQ, on the other hand, is a true-blue observability PaaS that helps you ingest log data from Kubernetes, on-prem servers or cloud VMs, applications, and several other data sources without a price shock. As LOGIQ uses S3 as the primary storage layer, you get better control and ownership over your data and as much as 10X reductions in costs in large-scale deployments. In this article that’s part of a two-article series, we’ll demonstrate how you can get started with Jenkins log analysis using LOGIQ. We’ll walk you through installing Logstash, setting up your Jenkins instance, and ingesting log data into LOGIQ to visualize and analyze your Jenkins logs.

Before you begin

Before we dive into the demo, here’s what you’d need in case you’d like to follow along and try integrating your Jenkins logs with LOGIQ:

A LOGIQ instance: If you don’t have access to a LOGIQ instance, you can quickly spin up the free-forever Community Edition of LOGIQ PaaS.
A Jenkins instance

Installing Logstash

Logstash is a free server-side data processing pipeline that ingests data from many sources, transforms it, and then sends it to your favourite stash. We’ll use Logstash as an intermediary between Jenkins and LOGIQ that grooms your Jenkins log data before being ingested by LOGIQ.

To install Logstash on your local (Ubuntu) machine, run the following commands in succession:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install logstash

For detailed instructions on installing Logstash on other OSs, refer to the official Logstash documentation.

Now that we’ve installed Logstash download the flatten configuration and place it in your desired directory. The flatten configuration helps structure data before ingestion into LOGIQ. Once you’ve downloaded the flatten configuration, use the following Logstash configuration to push your Jenkins logs to LOGIQ:

input {
  tcp {
    port => 12345
    codec => json
  }
}
output { stdout { codec => rubydebug } }
filter {
    split {
        field => "message"
    }
  mutate {
    add_field => { "cluster_id" => "JENKINS-LOGSTASH" }
    add_field => { "namespace" => "jenkins-ci-cd-1" }
    add_field => { "application" => "%{[data][fullProjectName]}" }
    add_field => { "proc_id" => "%{[data][displayName]}" }
  }
ruby {
        path => "/home/yourpath/flattenJSON.rb"
        script_params => { "field" => "data" }
    }
}
output {
  http {
        url => "http://<logiq-instance>/v1/json_batch"
        http_method => "post"
        format => "json_batch"
        content_type => "application/json"
        pool_max => 300
        pool_max_per_route => 100
       }
}

Note: Make sure you change the path in the configuration to the path where you downloaded the flatten configuration file. Also, remember to replace the LOGIQ endpoint with the endpoint of your LOGIQ instance. If you haven’t provisioned LOGIQ yet, you can do so by following one of our quickstart guides.

Setting up Jenkins

Now that we’ve got Logstash ready to go, let’s go ahead and configure Jenkins to use Logstash. For this demo, we’ve created two Jenkins pipeline jobs whose logs we’ll push to Logstash. You can use your own Jenkins logs when following along.

To push Jenkins logs to Logstash, we first need to install the Logstash plugin on Jenkins. To install Logstash, do the following:

Log on to your Jenkins instance.
Navigate to Manage Jenkins > Manage Plugins.
Search for Logstash under Available.
Once Logstash shows up, click Install without restart.

After installing Logstash, we’ll go ahead and configure and enable Jenkins to push logs to Logstash. To configure Jenkins, do the following:

Navigate to Manage Jenkins > Configure System.
Scroll down until you see Logstash.
Enter the Host name and Port.

Note: In this example, we’ve entered the IP address and port number of the local Ubuntu machine on which we installed Logstash. Ensure that you provide the IP address and port number of the machine where you’ve installed Logstash.

Your Jenkins instance is now ready to push logs to Logstash

Shipping logs to LOGIQ

We’ve got Jenkins ready to ship logs to Logstash and Logstash prepared to pick them up and groom them for ingestion into LOGIQ. Let’s go ahead and start Logstash from the installation folder (/usr/share/logstash) and pass the custom configuration file we prepared above using the following command:

/usr/share/logstash# bin/logstash -f /etc/logstash/logstash-sample.conf

That’s it! Your logging pipeline is up and running. Now when you head over to the Logs page on your LOGIQ dashboard, you’ll see all of your Jenkins logs that Logstash pushed to LOGIQ.

From here, you can create custom metrics from your logs, create events and alerts, and set up powerful dashboards that help visualize your log data.

This completes our overview on shipping and visualizing your Jenkins logs with LOGIQ. In a future article, we'll show you exactly how you can create powerful visualizations from your Jenkins logs. In the meanwhile, do drop a comment or reach out to us in case have any questions or would like to know more about how LOGIQ can bring in multi-dimensional observability to your applications and infrastructure and bring your log data to life.

Getting Started with the LOGIQ PaaS Community Edition

Ajit Chelat — Thu, 24 Jun 2021 14:30:30 +0000

If you’ve been looking for an inexpensive way to run your own observability stack while maintaining complete control over your data and its security, look no further. The LOGIQ PaaS Community Edition is officially live!

With the LOGIQ PaaS Community Edition, you can:

Self-host your observability stack on a cloud provider of your choice – public or private
Ingest up to 50GB of log data per day with unlimited data retention
Store your log data on any S3-compatible cloud provider via the built-in Minio S3 service
Ingest logs from Syslog, RSyslog, Logstash, Fluent, AWS Firelens, JSON, and plenty more
Run up to 4 ingest worker processes

You’ll also get access to all of the LOGIQ Enterprise Edition’s features along with Community Support, free forever.

What’s more? Deploying LOGIQ PaaS is ridiculously easy! This article will show you exactly how you can deploy the LOGIQ PaaS Community Edition on your Kubernetes cluster.

Before you begin

To get you up and running with the LOGIQ PaaS Community Edition quickly, we’ve made LOGIQ PaaS’ Kubernetes components available as Helm Charts. To deploy LOGIQ PaaS, you’ll need access to a Kubernetes cluster and Helm 3.

Before you start deploying LOGIQ PaaS, let’s run through a few quick steps to set up your environment correctly.

Add the LOGIQ Helm repository

Add LOGIQ’s Helm repository to your Helm repositories by running the following command.

helm repo add logiq-repo https://logiqai.github.io/helm-charts

The Helm repository you just added is named logiq-repo. Whenever you install charts from this repository, ensure that you use the repository name as the prefix in your install command, as shown below.

helm install <deployment_name> logiq-repo/<chart_name>

You can now search for the Helm charts available in the repository by running the following command.

helm search repo logiq-repo

Running this command displays a list of the available Helm charts along with their details, as shown below.

$ helm repo update
$ helm search repo logiq-repo
NAME                CHART VERSION    APP VERSION    DESCRIPTION
logiq-repo/logiq    2.2.11           2.1.11         LOGIQ Observability HELM chart for Kubernetes

If you’ve already added LOGIQ’s Helm repository in the past, you can update the repository by running the following command.

helm repo update

Create a namespace to deploy LOGIQ PaaS

Create a namespace where we’ll deploy LOGIQ PaaS by running the following command.

kubectl create namespace logiq

Running the command shown above creates a namespace named logiq. You can also name your namespace differently by replacing logiq with the name of your choice in the command above. In case you do, remember to use the same namespace for the rest of the instructions listed in this guide.

Important: Ensure that the name of the namespace is not more than 15 characters in length.

Prepare your Values file

Just as any other package deployed via Helm charts, you can configure your LOGIG PaaS deployment using a Values file. The Values file acts as the Helm chart’s API, giving it access to values to populate the Helm chart’s templates.

To give you a head start with configuring your LOGIQ deployment, we’ve provided sample values.yaml files for small, medium, and large clusters. You can use these files as a base for configuring your LOGIQ deployment. You can download these files from the following links.

values.small.yaml for small clusters.
values.medium.yaml for medium clusters.
values.large.yaml for large clusters.

You can pass the values.yaml file with the helm install command using the -f flag, as shown in the following example.

helm install logiq --namespace logiq --set global.persistence.storageClass=<storage_class_name> logiq-repo/logiq -f values.small.yaml

Read and accept the EULA

As a final step, you should read our End User’s License Agreement and accept its terms before you proceed with deploying LOGIQ PaaS.

Latest LOGIQ PaaS component versions

The following table lists the latest version tags for all LOGIQ components.

Image	Version
`logiq-flash`	2.1.11.27
`coffee`	2.1.17.4
`logiq` Helm chart	2.2.11

Install LOGIQ PaaS

Now that your environment is ready, you can proceed with installing LOGIQ PaaS in it. To install LOGIQ PaaS, run the following command.

helm install logiq --namespace logiq --set global.persistence.storageClass=<storage class name> logiq-repo/logiq

Running the above command installs LOGIQ PaaS and exposes its services and UI on the ingress’ IP address. Accessing the ingress’ IP address in a web browser of your choice takes you to the LOGIQ PaaS login screen, as shown in the following image.

If you haven’t changed any of the admin settings in the values.yaml file you used during deployment, you can log into the LOGIQ PaaS UI using the following default credentials.

Username: flash-admin@foo.com
Password: flash-password

Note: You can change the default login credentials after you’ve logged into the UI.

Your LOGIQ PaaS instance is now deployed and ready for use. Your LOGIQ instance enables you to ingest and tail logs, index and query log data, and provides search capabilities. Along with the LOGIQ UI, you can also access these features via LOGIQ’s CLI, logiqctl.

Now that you have full access to your very own LOGIQ PaaS instance, you should try using it to amplify your observability practices. You can use LOGIQ to observe your Kubernetes clusters, set up centralised observability for your CI/CD pipelines, monitor your applications and infrastructure, or even tail and analyse logs from AWS CloudWatch or other data sources – all without the pricing shock that the usual log management and analysis solutions provide.

Do drop a comment or reach out to us if you’d like to know more about how LOGIQ PaaS can help you deliver always-on applications and infrastructure at scale through efficient log management and analysis.

Run your favorite Helm Chart using MicroK8s in 5 minutes

Ajit Chelat — Wed, 23 Jun 2021 13:35:02 +0000

Helm is a Kubernetes package manager that helps you find, share, and use software built for Kubernetes. With Helm Charts, you can bundle Kubernetes deployments into a single package you can install by running a single command. At LOGIQ, we use Helm Charts on the regular. One of our most commonly used Helm Charts is logiq – the same Helm Chart we use for quick deployments of the LOGIQ observability platform for customers, prospects, and folks who’d love to know more about what we’re building.

This article will explain how you can deploy your favorite Helm Chart on MicroK8s in under 5 minutes.

What is MicroK8s?

MicroK8s is a lightweight, pure-upstream Kubernetes aiming to reduce entry barriers for K8s and cloud-native application development. It comes in a single package that installs a single-node (standalone) K8s cluster in under 60 seconds. While MicroK8s has all the Kubernetes core components, it is also opinionated, which means that many of the add-ons you would typically look for in Kubernetes, such as DNS, Helm, registry, storage, etc. are all a single command away.

What is LOGIQ?

LOGIQ is a complete observability platform for monitoring, log aggregation, and analytics with an infinite storage scale that aims to bring simple and powerful logging to the masses. LOGIQ uses AWS S3 (or S3-compatible storage) for data at rest and allows the sending of logs from Kubernetes, on-prem servers, or cloud VMs with ease.

The LOGIQ platform includes:

A User Interface (UI)
A command-line toolkit
A monitoring stack for time-series metrics, and;
A log analytics stack for log data.

Now that you’re acquainted with MicroK8s and LOGIQ and their awesomeness, why don’t we jump right into the integration?

The Setup

In this demo, we’ll use the logiq Helm Chart. You can also use your own favorite Helm Chart that you’d like to try out. Let’s also assume that you have access to the Linux operating system.

Installing MicroK8s

As a first step, let’s install MicroK8s on your machine by running the following commands:

sudo apt-get -y update
sudo snap install core
sudo snap install microk8s --classic
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
sudo microk8s config > ~/.kube/config

Now, let’s check whether MicroK8s is up and running or not with the command:

sudo microk8s status

Enabling add-ons

Now that we have MicroK8s up and running, let’s set up your cluster and enable the add-ons that MicroK8s readily provides, like Helm, DNS, ingress, storage, and private registry. These add-ons can be enabled and disabled at any time, and most are pre-configured to work without any additional setup.

Run the following commands to enable add-ons:

microk8s enable helm
microk8s enable storage
microk8s enable dns
microk8s enable ingress
microk8s enable registry
microk8s.kubectl config view > $HOME/.kube/config

Provisioning an IP address

We need an endpoint or an IP address to access the application we’re spinning up. This endpoint can either be within or outside our cluster. For this, let’s leverage MetalLB – a Kubernetes-aware solution that can monitor for services with the type LoadBalancer and assign them an IP address. Alternatively, you can also set an IP address while enabling add-ons.

While provisioning an IP address, you can use your local machine’s IP address, which pulls up the stack at IP-address:80. If you do not know your local machine’s IP address, run the ifconfig command as shown below and use the output of the command:

ifconfig:
wlp60s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.27 netmask 255.255.255.0 broadcast 192.168.1.255

Next, enable MetalLB by running:

microk8s enable metallb
Enabling MetalLB
Enter each IP address range delimited by comma (e.g. '10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111'): 192.168.1.27-192.168.1.27

Note: If you’re spinning up an EC2 instance from AWS, MetalLB might not work due to private/public IP configuration. We’ll take a closer look at and resolve this issue in another article.

Bring in the Helm Chart

Now that the configuration bits are in place, it’s time to bring in your Helm Chart. As mentioned above, we’re using the logiq Helm Chart and Helm 3 in the following commands. You can replace the Helm Chart repo URL in the following command with your own Helm Chart’s repo URL if you’re trying another chart.

Run:

helm repo add logiq-repo https://logiqai.github.io/helm-charts
helm repo update

Bringing up LOGIQ

Next, let’s create a namespace called logiq for the stack to spin up from and start running with the command:

microk8s kubectl create namespace logiq

And then run helm install with the storage class set to the microk8s-hostpath as shown below:

helm install logiq -n logiq --set global.persistence.storageClass=microk8s-hostpath logiq-repo/logiq -f values.yaml  --debug --timeout 10m

Note: The values.yml file used in the command above is customized to suit our cluster’s configuration. You can download the values.yml file from docs.logiq.ai and edit it to suit your cluster’s needs, and then run the above command.

LOGIQ is now ready to go! Before we get to the UI, let’s inspect the pods in your cluster by running the following command in the logiq namespace you created:

microk8s kubectl get pod -n logiq

We can now access the LOGIQ UI by hitting the MetalLB endpoint we defined earlier in this article. To find the endpoint, let’s search for the LoadBalancer service that knows which IP address MicroK8s exposes.

Run the command:

$microk8s kubectl get service -n logiq |grep -i loadbalancer
logiq-kubernetes-ingress  LoadBalancer   10.152.183.45  192.168.1.27

80:30537/TCP,20514:30222/TCP,24224:30909/TCP,24225:31991/TCP,2514:30800/TCP,3000:32680/TCP,514:32450/TCP,7514:30267/TCP,8081:30984/TCP,9998:31425/TCP     18m

Now, using a web browser of your choice, navigate to the IP address shown by the LoadBalancer service above: http://192.168.1.27:80

And voila! Our LOGIQ deployment using a Helm Chart is up and running!

On a side note, we wanted to talk about how cumbersome logging is. Most of the logging solutions out there hold data in proprietary databases that use disks and volumes to store their data. What’s wrong with that, you ask? Well, disks and volumes need to be monitored, managed, replicated, and sized. Throw in clustering in the mix and your log analytics project is now working on someone else’s software and is a storage nightmare. For these reasons, we built the LOGIQ observability platform – to bring easy logging to everyone who needs it. Using LOGIQ, you can ingest log data from Kubernetes, on-prem servers or cloud VMs, applications, and several other data sources, helping you gain complete visibility over your infrastructure and applications without burning a massive hole in your pocket.

We love making logging easy. To show you how easy it is to get going with the LOGIQ platform, our next article will show you how to automate the LOGIQ platform’s deployment on AWS using Helm Charts on MicroK8s using a CloudFormation template. Meanwhile, if you have questions about LOGIQ and would like to know more, please do leave a comment below or visit our website and reach out!

Shift from API Monitoring to API Observability with LOGIQ

Ajit Chelat — Tue, 22 Jun 2021 15:08:16 +0000

APIs – by now, we’re all familiar with the term. Every service or software we use or build today either uses or is an API. If APIs are a central pillar in your building and delivery of software and services, you’ll know that the success of your software or services depends on the integrity, availability, and performance of your APIs. Traditional API monitoring sure does help you stay on top of uptime, security, and performance. Still, it is limited to being a black box form of monitoring – you’re only testing API behavior that you’d only see externally. You’d still have to guess what’s causing issues with your APIs as you’re only testing, measuring, and monitoring them against metrics you already know – like request rates, error rates, or status codes.

Traditional API monitoring lets you monitor system health and performance but can’t help you identify and troubleshoot what’s causing issues. And the more your system’s dependency on APIs increases, the more you’ll find yourself modifying the metrics you track, guessing which parts of your API may cause problems, and not moving any closer to root causes.

At this stage, establishing a proper API Observability strategy starts making more sense. API Observability is all about making your APIs more observable. Instead of relying on predetermined metrics and monitoring and waiting for failure, API Observability lets you dive into the unknown unknowns of your APIs by observing how they work internally. With API Observability, you can analyze data exposed by the internals of your API system and identify patterns and behavior that help you prevent threats, identify and troubleshoot issues, and understand API usage.

What’s the easiest way to make your APIs observable, you ask? Use LOGIQ.

LOGIQ is a complete observability platform for monitoring, log aggregation, and analytics with an infinite storage scale. The LOGIQ platform includes:

A User Interface (UI)
A command-line toolkit
A monitoring stack for time-series metrics, and
A log analytics stack for log data.

But how does LOGIQ help with API Observability? Here’s how.

Support for popular API Gateways

LOGIQ can aggregate logs from popular API gateways like Istio, NGINX, HAProxy, Apache, and more. Integrating LOGIQ with your API gateways gives you complete visibility into API usage in your environment.

Automated extraction for API attributes

LOGIQ includes full support for GROK expressions. You can write powerful rules in GROK to extract API attributes from standard API log formats such as Common Log and Apache. With GROK expressions, you can comb through API log traces to extract information like API methods, response times, URLs, sender information, payload length, request rates, and error rates.

Visualizing API data with LOG2Metrics

With LOG2Metrics, you can transform ingested API log traces into multiple time-series visualizations using powerful attribute or pattern-based group-by expressions. For example, you can easily query for GET requests made to the endpoint /v1/resource/<id> and generate time-series visualizations by ID.

Performance Visualization & Alerting

Using LOGIQ, you can plot API response times extracted from your API logs and gain total visibility over how your APIs perform. Moreover, LOGIQ’s alerting capabilities let you build alerts that notify you instantly when an API begins to underperform. You can also eliminate false positives by creating alerting rules that only trigger after being validated against frequency thresholds in intervals that you can customize. For example, you could choose to get alerted if an event occurs more than 10 times in a 5-minute interval.

Historical Reporting

You can generate insightful and periodical ad hoc reports on historical API data scheduled with a built-in CRON job. For example, with a few clicks, you can create a report that shows you all client IP addresses that generated 4xx errors, grouped by HTTP status codes and IP addresses.

Extract Business Intelligence from APIs

Empower your Support and Services organizations by using LOGIQ to extract insightful business-level metrics around API usage, avenues to harden security, and product analytics from log data. For example, in case your software or service offers public-facing APIs and you’d like to know how your partner integrations are faring, you can monitor the health of your API usage by partners and notify your support and services team when you start seeing increased error rates.

Closing Thoughts

An API Observability platform is more than an accessory for your overall API strategy. The right approach to API Observability can provide you with invaluable insights about API usage while identifying security loopholes and maintaining the overall health, performance, and availability of your APIs. Now’s the time to adopt API Observability to optimize the way your APIs are shipped and how they perform by establishing visibility into the inner workings of your API system. To get started with API Observability, install the free-forever LOGIG PaaS Community Edition and integrate it with your API server.