DEV Community: CloudFabrix

Robotic Data Automation (RDA): Reducing Costs and Improving Efficiencies of Your Log Management Investment

CloudFabrix — Tue, 14 Sep 2021 13:31:29 +0000

People’s involvement has been inevitable with log management despite advancements in ITOps. Log management at a high level collects and indexes all your application and system log files so that you can search through them quickly. It also lets you define rules based on log patterns so that you can get alerts when an anomaly occurs.
Log management analytics solution leveraging RDA has been able to detect anomalies and aid predictive models over a machine learning layer. This has demonstrated improved efficiencies and a direct reduction in costs.

How does a conventional log management system work?

To make things better, log analytics has been deployed. However, log analytics alone cannot replace human interactions. Log analytics systems produce a significant increase in alerts which trigger the need for more support from IT personnel, thus limiting its progress anytime soon. There is still a need to blend it with traditional operations practices, namely on-call monitoring teams and consulting analyst teams who respond to any alert received by the system.

However, none of these mechanisms could explain or ease up the burden of an engineer sitting at his workstation while he/she responds to every notification generated by the log monitoring tools without being able to pre-determine whether such notifications are critical or not.

Challenges of a conventional log management tool

A log management tool has to be fast. Without this, users will not be able to search for logs. It also has to have enough storage so that it can index logs from an entire enterprise. This can be very expensive depending on the number of servers or applications you need information from.
A log management tool needs to support every platform (Windows, Linux, Unix) your business uses in order to pull relevant metrics and log data you need for troubleshooting and analysis.
In a conventional way, automating aggregation and distributing to different systems is not possible.

RDA-enhanced log management system

With Robotic Data Automation(RDA), your log management system will be augmented with an intelligent layer to increase efficiency without disrupting any established processes. RDA also enables the creation of advanced ML-based dynamic baselining and predictive models that makes anomaly detection efficient and more reliable.

The benefits of using an RDA-enabled log management solution for your business include reduced costs and improved efficiencies in managing logs.

Automates alerts: The first benefit is that RDA automates the alert generation process from manual rules-based. It analyzes the raw log data and dynamically learns alert conditions by minimizing the risk of missing alerts from new/modified log entries.
Auto-correlation of alerts: It supports intelligent alerting capabilities with auto-correlation of alerts across different systems/services, resulting in a reduced number of alerts turning into trouble tickets.
Contextual Insights: It automatically provides contextual log data when analyzing the trouble ticket thereby reducing the need for users to manually search logs
Removes duplicates: It prevents unnecessary alert noise by removing duplicates across various systems while increasing the overall quality of alerts and insights derived from them.
Alert log pattern summary: It automatically summarizes the alert log patterns as time-series data making it efficient and cost-effective to retain historical knowledge.
Builds regression pipeline: It allows building a regression pipeline on any attribute for predicting the trend and anomalies.
Brings new data sources: RDA also helps in bringing new data sources into existing log management systems where native methods are either complex or simply do not exist.

RDA offers out-of-box integration with Splunk and Elastic search products. Combining RDA with Splunk/ELK not only significantly reduces the overall efforts required for managing IT Operations but also provides an excellent set of capabilities for detecting anomalies in near real-time. Also, using RDA reduces false positives, meaning alerts generated are not a result of false or random events.

By building these capabilities on top of existing log management analytics investments, organizations and IT Ops can reduce costs while improving efficiency and business outcomes.

RDA vs AIOps

If you are looking at optimizing the operations of a single or small subset of IT tools like a log management analytics platform, RDA can be a better choice. Standalone RDA is faster to deploy/adopt and validate the benefits at a lower cost.

AIOps, on the other hand, is designed to bring transformation across IT functions. It integrates with multiple data sources and performs cross-domain correlation and analysis to deliver actionable insights.

RDA with CloudFabrix

The CloudFabrix AIOps platform and all of its products are pre-packaged with RDA. This includes Operational Intelligence, Asset Intelligence, and Observability-In-a-Box.

RDA is also provided as a standalone tool and as a hosted offering to help iterate, author, and publish data workflows.

So with CloudFabrix, organizations can start with RDA, validate the technology promise, and seamlessly upgrade to AIOps, delivering the value across functions.

Excessive Alert Noise: Cause, effect, and solution

CloudFabrix — Thu, 24 Jun 2021 10:52:59 +0000

With an exponential growth in the IT sector over the last few years, traditional operational tools and process isn’t enough to stay ahead of the market. Problems/anomalies are treated as ‘events’. Each of these events triggers an alert in the system leading to separate incidents that require individual resolution. With an increase in data, hybridization, operational tools, countless metrics, there has been a corresponding increase in alert volume. This causes inundation of high volume and variety of log data, usually with multiple false and redundant alerts.

About 40% of IT organizations see over a million event alerts a day, with 11% receiving over 10 million alerts a day.

Most IT teams today operate in disparate silos, often unaware of the assets they have, their utilization or inter-dependence thereby compounding the problem.

Why is there an excess of alert noise?

Some of the common reasons for an increasing volume of alert noise are:

Lack of stack awareness
Static thresholds
Alert Storms

Lack of stack awareness

Traditional legacy systems process this differently using approaches that solely rely on signature/footprint matching. This does not allow for Machine Learning capabilities to perform impact analysis and correlation of alerts/events from multiple stack elements.

Static thresholds

Static thresholds are unable to take into account the dynamic nature of IT workloads. This creates alerts at pre-established levels, that no longer works for a majority of the workloads leading to an excessive number of alerts. Being unable to identify and create contextual awareness of where to disabled alerts and where to increase alert capacity proves to be a barrier.

Alert Storms

Outages both planned and unplanned stir up alert storms. Network disruption causes employees, remote users, and devices to disconnect leading to a high volume of unwanted alerts.

Alert Noise is estimated to cost an average of $1.27Million per year to companies.

How does CloudFabrix help with alert reduction?

AIOps solution has been implemented by 60% of organizations to reduce noise alerts and identify real-time root cause analysis.

The CloudFabrix AIOps platform uses combination of user configurations and advanced AI/ML algorithms such as correlation , anomaly, forecasting etc to reduce alert volume through grouping, suppression and prevention.

Rule Based → AI/ML and Analytics Based Approach

Instead of relying on manual tagging and rule based grouping, CloudFabrix uses time based and asset dependency based automated grouping of multiple alerts into actionable problems. It further uses predictive analytics thereby reducing alert noise by a significant number.

Static Thresholds → Dynamic Thresholds

Static thresholds ignore dynamic nature of IT workloads and create alerts at per-established levels, which won’t work for the majority of the IT workloads that are dynamic in nature. This results in excessive number of alerts.

To address this problem

Granular Controls: Provide granular alert controls to tune telemetry collection interval. And to minimize the alerts caused to metric fluctuations we provide hi-watermark, lo-watermark and minimum occurrence controls.
Dynamic Thresholds: Dynamic thresholds establish a baseline for every metric and raise an alert only if the metric is deviating from baseline.
Identify heavily utilized assets where alerting should be disabled or more capacity should be added.

Alert Storms → Actionable Incidents

Alert Storms can occur anytime, but more so during unplanned outages , planned outages and cascading alerts

Planned Outages: With our platform, alerts can be configured to be ignored during planned outages like patching, backup or maintenance. In addition to this, we are able to automatically exclude network device access ports from monitoring, as this can cause an excessive number of unwanted alerts, whenever employees, remote users, phones etc. connect/disconnect from the network.
Unplanned Outages: and device fluctuations or flapping situations cause alert storms, which we detect automatically and suppress the alerts during unplanned outages like network disruption or device unavailable events.
Cascading Alerts: this happens when a device/component fails resulting in alerts from other parts due to interdependence or lost connectivity between the monitoring system and the dependent devices. These deluge of alerts are often pointing to the same underlying issue. These sort of alerts can be grouped together if the system has knowledge of the interdependencies and can identify the underlying root cause issue.

Please feel free to ask anything about aiops and we will be happy to answer any queries you have.

Taming the Data Problem and Accelerating AIOps implementations with Robotic Data Automation (RDA)

CloudFabrix — Fri, 30 Apr 2021 13:08:48 +0000

What is Robotic Data Automation (RDA)

Robotic Data Automation (RDA) is a new paradigm to help automate data integration and data preparation activities involved in dealing with machine data for Analytics and AI/Machine Learning applications. RDA is not just a framework, but also includes a set of technologies and product capabilities that help implement the data automation.

RDA enables enterprises to operationalize machine data at scale to drive AI & analytics-driven decisions.

RDA has broad applicability within the enterprise realm, and to begin with, CloudFabrix took the RDA framework and applied it to solve AIOps problems – to help simplify and accelerate AIOps implementations and make them more open and extensible.

RDA automates repetitive data integration, cleaning, verification, shaping, enrichment, and transformation activities using data bots that are invoked to work in succession in “no-code” data workflows or pipelines. RDA helps to move data in and out of AIOps systems easily, thereby simplifying, and accelerating AIOps implementations that otherwise would depend on numerous manual data integrations and professional services activities.

To work with RDA workflows, CloudFabrix provides a collaborative integrated development environment (IDE) called AIOps Studio, which is an integral part of all CloudFabrix solutions. Using AIOps Studio, developers, services, and implementation personnel can design, author, iterate and publish data workflows and finalized workflows can then be published to production systems for the workflows to take effect.

Why RDA is Needed?

Artificial Intelligence for IT Operations (AIOps) requires processing vast amounts of data obtained from various hybrid IT data sources, that are spread across on-premises, cloud, and edge environments. This data comes in various formats and delivery modes. Additionally, results and outcomes of such data processing need to be also exchanged with other tools in the IT ecosystem (Ex: ITSM/Closed-loop automation/Collaboration Tools and BI/Reporting tools).

All of this requires integrating, ingesting, preparing, verifying, cleaning, transforming, shaping, analyzing, and moving data in and out of AIOps systems in an efficient, reusable, and scalable manner. These essential tasks are most often overlooked in AIOps implementations and cause significant delays and increase costs of AIOps projects.

Challenges

Let us understand what some of the key challenges in data preparation & data integration activities are, when implementing AIOps projects.

Different data formats (text/binary/json/XML/CSV), data delivery modes (streaming, batch, bulk, notifications), programmatic interfaces (APIs/Web development/Webhooks/Queries/CLIs)
Complex data preparation activities involving integrity checks, cleaning, transforming, and shaping the data (aggregating/filtering/sorting)
Raw data often lacks application or service context, requiring real-time data enrichment bringing in context from external systems
Implementing data workflows require specialized programming/data science skill set
Changes in source or destination systems require rewriting/updating connectors

Traditional Approach of Data Handling in AIOps:

In traditional approach, AIOps vendors provide a set of out-of-the-box integrations and once you connect AIOps software to your data sources, you are now pretty much at the mercy of how your data gets utilized, processed for producing results & Outcomes.

Black box approach of data acquisition, processing, and integration
Use cases and scenarios limited to what the platform supports
Integrations mostly predefined/hard coded limiting reuse
Complex scripting modules or cookbooks requiring specialized/programming skills (Javscript, Python etc.)
Difficult to bring in external integrations for intermittent data processing (ex: enrichment)
Difficult to access data in a programmatic way for complementary functions (ex: data access for scripting, reporting, dashboarding, automation etc.)

These are all inhibitors to effective AIOps implementations by way of adding delays & costs (manual data prep/handling activities)

Need of the Hour: Robotic Data Automation for AIOps

Robotic Data Automation (RDA), a key enabler for AIOps 2.0

RDA automates DataOps, similar to what RPA did to automate business processes. RDA is integral part of AIOps solution that provides augmented data preparation and integration capabilities. RDA is both a data automation framework and a toolkit to accelerate and simplify all data handling in AIOps implementations. Highlights

Implement No-code Data Pipelines using Data bots
Native AI/ML bots
CFXQL – Uniform Query Language
Inline Data Mapping
Data Integrity Checks
Data masking, redaction, and encryption
Data Shaping: Aggregation/Filtering/Sorting
Data Extraction/Metrics Harvesting
Synthetic Data generation

Benefits

Simplify and Accelerate AIOps implementations
Reduces time/effort/costs tied to data prep and integrations.
Suitable for DevOps/ProdOps personnel (no need of data scientist skills)

Example Use Cases and Scenarios

Log Clustering: Ingest app logs from cloud and on-prem, run ML models to cluster logs, push results to Kibana/CFX dashboards
CMDB Synchronization: Take latest asset inventory from CFX and push it to CMDB.
E-Bonding of tickets from partner/subsidiary ITSM to customer’s ITSM (Ex: BMC incidents to ServiceNow)
Incident NLP Classification: Ingest tickets from ServiceNow, do NLP classification with OpenAI (GPT-3) and enrich tickets back in ServiceNow
Anomaly Detection: From Prometheus (or any monitoring tool), get historical CPU usage data for a node (Hourly). Apply regression and send a message on Slack with a list of anomalies as an attachment.
Ticket Clustering: Take last 24-hrs incidents from ServiceNow, apply clustering on tickets and push results to a new dataset for visualization in CFX dashboard.
Change Detection: Take baseline inventory of AWS EC2 VMs and compare against current state to highlight unplanned changes.

How it Works

CloudFabrix AIOps platform and all its products (Operational Intelligence, Asset Intelligence, Observability-In-a-Box) are pre-packaged with RDA. Additionally, RDA is also provided as a standalone tool and as a hosted offering to help iterate, author, and publish data workflows.
To begin with, RDA provides a lot of out-of-the-box data integrations (called extensions), which are written in Python. Customers can use extensions to connect with their data sources and access them through tags or data sets. All DataOps activities like performing integrity checks, data cleaning, transforming, and shaping the data (aggregating/filtering/sorting) can be performed with data bots invoked in a pipeline, which is essentially a series of data processing steps, that typically take data from a source, do some processing and transformation, and send data to a sink. A unique aspect here is these are “No-Code” pipelines, meaning you do not have to know Python or JavaScript to implement these pipelines, and the pipeline just uses configurational semantics using text that is similar to natural language.
Following is an example of data workflow that takes AWS EC2 Inventory Baseline

@c:simple-loop loop_var = "aws-prod:us-east-1,aws-dev:ap-south-1,aws-dev:us-west-2"
--> *${loop_var}:ec2:instances
--> @dm :save name = "temp-aws-instances-${loop_var}"

--> @c:new-block
--> @dm :concat names = "temp-aws-instances-.*"
--> @dm :selectcolumns include = 'InstanceId|InstanceType|State.Name$|LaunchTime|^Placement.AvailabilityZone$'
--> @dm :fixcolumns
--> @dm :save name = 'all-aws-instances-Mar-25'

Fig: AIOps Studio: Visualization of Data Pipeline execution

AIOps product has a lot of prebuilt pipelines that provide a turnkey experience for all core AIOps use cases and scenarios, so that customers perform integrations with their tools and get to work right away. Anytime customizations are needed, either to address new use case or scenario, or update the way data transformations or enrichments are done, customers can use RDA utility to iterate and experiment with pipelines, and once pipelines are functioning as expected, customers can then publish the pipelines to AIOps platform to be deployed into production. Resultant datasets can be easily visualized in CFX dashboards.
This process is like how developers use Integrated Development Environment (IDEs) to write code, iterate, compile and build images, which are then pushed onto a runtime environment. In the same way, RDA provides a collaborative Jupyter style notebook authoring tool and workflow visualization tool, using which pipelines can be built or customized, and once finalized they can be pushed on to the production AIOps platform. For workflow authoring, RDA is also available as a standalone tool that can be deployed in a customer’s own environment or it can be used as a service within CloudFabrix’s cloud-hosted environment.

Fig: AIOps Studio provides IDE-like capabilities for DataOps
Getting Started – AIOps Studio
It is very easy to get started with RDA using AIOps Studio, which is visual IDE for authoring and publishing data pipelines. We have few options available.

Install on top of Docker
AWS image
Hosted service

Once you bring up RDA install using one of above options, you can invoke AIOps Studio and pretty much do all data operations using the studio, that provides intuitive user interface.

Fig: AIOps Studio Welcome Page

AIOps Studio Tasks

Following are some of the tasks that can be performed from AIOps Studio:

Add a New Pipeline
Edit a Pipeline
Execute a Pipeline
Pipeline Syntax Verification
Inspect Pipeline Data Results at Each Stage
Visualize and Debug Pipeline Execution
Publish Pipeline to Production App
Explore Data Automation Bots
Explore API models for Data Automation Bots
Explore Datasets
Administration: Manage Bot Sources
Administration: Check Connectivity for Configured Sources
Administration: View Plugins
Administration: Add New Plugin (coming soon)
Show Inline Help
More … Anyone interested in trying out RDA can sign up for interest list here: https://www.roboticdata.ai/signup/

Observability is Transforming ITOM Landscape as Next Generation Monitoring

CloudFabrix — Thu, 15 Apr 2021 05:45:20 +0000

First things first. Observability is inherent as a principle to a system and not something that is instilled. Here, we are addressing observability as an open source-based solution in the context of insightful monitoring within the ITOM landscape.

ITOM is now in the middle of addressing the needs of the expanding and dynamic nature of IT infrastructure as a function. It is no longer about being a monolithic computing stack. It is now beyond monitoring discrete infrastructure elements.

Simple systems are smart enough to run a diagnosis and correct it on their own. But, native-cloud businesses with micro service architecture and distributed systems are complex.

System admins and ITops analysts were the eyes of systems, services and apps. It circled around ticketing, resolution, and people dependency. Complex systems depending on people increased the level of failure.

Now, ITOM is about observability that creates a proactive and transparent among complex and modern ecosystems.

Why Observability for your ITOM?

Observability provides a deeper operational visibility that brings an organization-wide sense of customer experience. For digital transformation and cloud adoption, observability is one of the essential and critical pillars. The monitoring workload is also shifted to development. Apart from these, a few trends make observability a necessity for complex structures.

Businesses are living with the pressure to innovate fast. The development teams launch something every other hour, and they need to talk to internal systems for insights.
Apps and software have to deliver a high customer experience.
To meet these business needs, technology stacks is adding more and more tools. It is getting bigger and complex.
To put all these in place, there is a lot of demand for devops and automation. Creating intelligent autonomous systems is the ITOM’s new skill to be added.

Observability has been able to respond to these trends. Teams love it: Observability is gaining attention in the software world. It is able to effectively enable engineers to deliver excellent customer experiences with software despite the complexity of the modern digital enterprise. It is helping modern software teams. On the whole, it have proven to:

Deliver high-quality software at scale
Build a sustainable culture of innovation
Optimize investments in cloud and modern tools
See the real-time performance of their digital business

Observability, across different use cases is able to provide full visibility into rapidly scaling infrastructure and applications, anomaly detection and metrics. It is not yet another monitoring tool.

Observability, the future of systems and app monitoring

Monitoring as a tool has detected problems and anomalies in applications. Troubleshooters used to also gain insights into capacity requirements and performance trends over time. But, monitoring would be useless if these systems are not externalized enough.

Observability as a next generation monitoring mechanism measures how a system’s internal state can infer external outputs. It develops organizational capabilities of monitoring and analyzing events, along with KPIs and other data. Observability yields actionable insights.

CloudFabrix’s Observability solution is built using open source and open telemetry components. It draws data from across different data streams coming from servers, databases, applications, tools and services through turn-key integrations across the full DevOps stack. It also includes a lightweight agent deployed that gets automatically provisioned on the host operating systems.

Observability is applied in the following use cases:

To reduce visibility gaps for modern applications architecture and dynamic workloads since the existing monitoring tools are not designed to support.
Provide context to the monitoring data to explain why and where questions.

Get a full tutorial of observability-in-a-box.

Observability in a box in the AIOps context

With growing complexity, the above use cases hint the need for inclusion of observability with AIOps.

Traditional AIOps could not be applied for all use cases that challenged ITOM landscape. It served the immediate purpose but with high investment of resources and longer time to value.

Observability in a box in the AIOps context
With growing complexity, the above use cases hint the need for inclusion of observability with AIOps.

Traditional AIOps could not be applied for all use cases that challenged ITOM landscape. It served the immediate purpose but with high investment of resources and longer time to value.

AIOps 2.0 with observability in a box provides quick turnaround with easy implementation. It brings an outcome-driven approach for metrics, events, logs, traces, and alerts.

Traditional ITOM comes with data centers geographically distributed. This adds to the alert noise and ticket volume. The problem resolution time is extended with reactive operations and lack of ability to preempt outages.

AIOps 2.0 by CloudFabrix with observability enables:

Close identification of specific problems in a switch.
Dependency recognition, where the problem has actually triggered from.
Identification of other observability gaps
Asset intelligence
Reduced MTTR (mean time to repair)

It just gets better with more trained data and resolution models. It uses asset and model intelligence to correlate alerts where connected data relies on feeding information. Recommendations about adding new servers reduces the workload while leveraging the capabilities of Edge AI.

Why Observability is key for AIOps journey?

Observability is about providing a unified view across IT operations and systems which is tied to your business KPIs. AIOps is a part of Observability. AIOps 2.0 is all about intelligence. This element of intelligence is impossible without the insights drawn by observability.

We are referring to:

Cost reduction recommendations rather than matrices analysis
Delay avoidance rather than processing delay intimation
Real-time metric data analysis and trends with aligned actionable insights and anomaly detection.

So, rather than telling you where the problem is, AIOps solution with observability will tell you where the problem is, how it is solved, and what should be done to avoid it later. This continuous feed of intelligence is only possible when AIOps come together with observability.
Click here to learn more about CloudFabrix with the help of this dynamic creative.