DEV Community: Sabari Rohith

How to get Azure Data Factory Pipeline Failure Notification?

Sabari Rohith — Fri, 23 Jun 2023 15:01:00 +0000

What is Azure Data Factory Pipeline?

Azure Data Factory is a cloud-based data integration tool focusing on data extraction, transformation, and loading. A pipeline in Azure Data Factory is a collection of processes that move data to a shared repository, such as a data warehouse.

Why it is important to monitor Azure Data Factory pipeline failures?

An Azure Data Factory pipeline is an orchestration that automates numerous tasks related to data extraction, transformation, and loading; therefore, if any of the events fail, the entire pipeline will fail.

Organizations depending on Data Factory pipelines will have a hassle-free experience if they keep track of the failed pipeline runs in the Data Factory and rectify it.

How to monitor failed pipeline runs in Data Factory?

Azure Data Factory pipeline run history can be accessed in the Studio of the respective Factory. The runs, inputs, outputs, and failure details will be available for each run.

Failed runs in a Data Factory Pipeline can be monitored by navigating to Monitor -> Alerts & metrics. You will need to identify the monitoring criteria to define the alert logic and evaluation period.

You can set up notification channels to get alerts and stay informed on violations. Azure Action groups enable you to group several notification channels and instantly notify failure alerts to multiple channels.

Note: The defined criteria will only apply to the pipelines in a single Data Factory.

The image shown below displays a sample alert triggered using the alert rule configured in the Data Factory:

Data Visualization

Data Factory supports a wide range of metrics to understand performance, reliability, and availability. By default, the metrics are available at the level of a Data Factory. You can apply filters to visualize the data from a specific pipeline.

With the aid of visualization, you can configure new alert rules for monitoring at the pipeline level.

Challenges faced by Azure for pipeline monitoring

Monitoring Data Factory pipelines using Azure can present quite a few challenges:

It is difficult for someone without expertise in Data Factory to understand the relationship between the factory and pipelines and how to manage and monitor them
Most support cases are going to escalate to your Data Factory expert resulting in increased costs and a longer time to resolution
It takes much effort to configure the monitoring Managing multiple pipelines from different data factories is complicated

Azure Data Factory Pipeline monitoring in Serverless360

Serverless360 is a robust pipeline monitoring tool that can be handled by anyone, even those with limited product knowledge. In addition, Serverless360 has an effective support team that responds almost instantly to customer questions and continues tracking customer feedback.

Serverless360 offers Azure Data Factory monitoring, allowing you to easily monitor multiple pipelines from various data factories at one location. This centralized monitoring allows improved visualization, easy access to data, and real-time insights. It simplifies troubleshooting issues, reducing the efforts spent monitoring multiple pipelines from different factories.

Business Application

Business Application is a logical container that groups Azure services, comprising a line of a business solution, and enables centralized management and monitoring.

Monitoring Profiles

A Monitoring Profile is a customizable collection of monitoring rules that can be used to set up monitoring for Azure services associated with a Business Application.

Problem statement

A business orchestration that utilizes many pipelines can be challenging to map to the business function they are intended to perform. Business Applications make it possible to model a business environment and group pipelines together to make a logical container called a Business Application, which processes business data (e.g., data concerning a company’s engineering team). This allows you to manage the business function rather than the entire data factory.

Imagine the following scenario: You are responsible for managing and monitoring more than 50 pipelines that contribute to data related to the Recruitment process and more than 100 pipelines that contribute to the Onboarding process managed by the HR team at the company. It is essential to ensure that every pipeline operates properly because any failure in one pipeline activity fails the entire pipeline.

Solution

A combined operation of Business Applications and Monitoring Profiles in Serverless360 will help you implement what you need in this scenario.

The first step is to create two Business Applications that represent the different processes of the HR department, as discussed above: Recruitment and Onboarding.

Associate the necessary pipelines in the respective Business Applications. After creating the Business Application, you can use the Add option in the Resources section to associate pipelines.

The next step is creating and applying Monitoring Profiles for the Business Applications representing the Recruitment and Onboarding process.

The image below illustrates the profile configuration for the Recruitment process, where the Warning and Error thresholds for the Failed runs metric are set for a higher value:

The image below illustrates the profile configuration for the Onboarding process. Here the Warning and Error thresholds for the Failed runs metric are set for a minimal value:

Once these profiles are created, you must apply them to the corresponding Business Applications. A Monitoring Profile can be applied using the following path in a Business Application: Monitoring -> Profile settings -> Apply profile.

Monitoring will be initiated after profiles are applied, and you will be notified of failures instantly. Following is how the status of each Business Application is represented in Serverless360. You can navigate into the Business Application, which has errors, and act on it.

Clicking on the health state next to a pipeline will reveal the monitoring details.

Alert Escalation

Escalation Policies can be set up at the Monitoring Profile level. Each escalation time frame can use different notification channels, such as Slack, Service Now, and Pager Duty, to transmit alerts.

Manually resolve violations

Each violation alert triggered from Serverless360 contains a link to the violated resources in the respective Business Application, which will navigate you to the appropriate resource in Serverless360.

The pipeline can be manually rerun to resolve the violation after navigating the violated resource.

Automatically resolve violations

In addition to failure monitoring, you can specify an action to be taken in case of a rule violation. The combined event of an automated action with a rule violation saves the time taken over manual intervention and improves business continuity.

For example, suppose you have ‘n’ failed activity runs in the past few hours and want to rerun them when the n value reaches the configured threshold. A task can be configured to rerun failed pipeline runs in a specific time frame and mapped to the rule violation.

Let’s put it all together

Microsoft Azure provides pipeline monitoring via Azure Monitor by configuring a set of rules and alert logic shared by all pipelines available in an Azure Data Factory. The alert logic helps to detect data quality issues and performance bottlenecks. Managing multiple pipelines from different data factories requires repeated alert configurations multiple times, making it challenging to track the alerts.

By combining pipelines from multiple data factories into a logical container and using Monitoring Profiles to track the failures, Serverless360 increases its edge over Microsoft Azure. This helps to identify any issues quickly and proactively take corrective action, reducing downtime and increasing productivity. Having features like Automated Reruns in case of violations will be very useful for keeping pipelines intact.

To sum up, Serverless360 is a mature product to monitor and manage Data Factory pipelines. This allows you to delegate support from Azure experts to IT operations, lowering support costs and freeing up resources to work on delivering new solutions.

Try Serverless360 for free!

Configuring Azure Logic App Failure Alerts To Stay Ahead

Sabari Rohith — Thu, 25 May 2023 09:58:03 +0000

Azure Logic Apps is a cloud-based service provided by Microsoft Azure that allows users to create and run automated workflows. A trigger is the first step of a workflow that specifies the condition for running further steps in that workflow. Azure Logic Apps creates a workflow run each time the trigger fires successfully.

The details of each run, including the status, inputs, and outputs of each step of the workflow instance, can be accessed in the run history section of the Logic App. Each run can either execute successfully or fail due to some reasons.

Why is it essential to track Logic App failures?

Like any technology, Logic Apps can experience failures, which can cause serious consequences such as data loss, and disrupted business flows.

Take a sample scenario where the Logic App orchestrates a credit card payment. An HTTP request triggers the Logic App with payment details from the user, such as the card number, expiration date, billing address, etc. Then it uses a connector to authenticate the payment information. Once the information is authenticated, Logic App processes the payment using a payment gateway. Once the payment is successful, the Logic App updates the database and sends a confirmation email or text message to the customer indicating that the payment has been successfully processed.

In such a scenario, if there are any errors or exceptions during the payment processing, it is crucial that the Logic App can track these failures and quickly alert the relevant teams to mitigate the issue.

Azure Logic Apps provides capabilities to track and handle failures. The logic app failure alerts can be various channels like email, SMS, or other communication channels.

Configuring alert rules in the Azure portal

Alert rules can be created from the Monitoring section of the Logic App from the Azure portal. Azure provides an extensive list of metrics representing the critical aspects of the resources. Metrics and their respective thresholds can be configured with the alert rules.

Challenges Monitoring Logic Apps

Azure Monitor is handy when monitoring the resource as an individual entity. When there is a requirement to monitor the resources constituting a business flow as an application, it isn’t easy to implement monitoring which will help a support user understand the role of the Logic App in the broader context. The alerts received through Azure monitor are at the resource level. When multiple Logic Apps are monitored, it is challenging to track the alerts.

In such cases, Azure Monitor may only meet some of your needs. Serverless360 provides features that will help you monitor and manage Logic Apps in the real world.

How to monitor Logic Apps failures using Serverless360

Serverless360 is a cloud-based platform designed to allow users to manage and monitor their applications running on the Microsoft Azure cloud platform. It offers much tooling that allows users to monitor, troubleshoot, and manage serverless applications.

Setting up Azure Logic App monitoring using Serverless360 is straightforward and can be achieved by Business Applications. It is a logical container that groups a particular application’s resources.

A Business Application can be created by adding the required Logic Apps. In addition, various resources of different types constituting a business flow can be added.

Monitoring profiles allow users to configure monitoring rules for multiple resources of the same type or different types. Instead of configuring the monitoring rules at each resource level, a monitoring profile to monitor the Logic App failures can be created and applied to the Business Application, which will monitor all the Logic Apps in it.

When applying the monitoring profile to a Business Application, users can opt to automatically apply the profile to any other Logic App that will be added to the Business Application.

As soon as the resources are added to the Business Application in which the monitoring profile is applied, the resources get automatically monitored, and the status of the resources will be updated as below.

In some cases, more than monitoring the metrics of the Logic App is required, as there may be a slight delay for the metrics to get emitted in Azure. Serverless360 can monitor the failures in such cases by investigating the actual Logic App runs.

Resolving failures

In addition to alerting the failures, Serverless360 can resolve them by resubmitting the failed runs. Even though resubmitting a run is possible in the Azure portal, the challenge is identifying the already resubmitted runs. Serverless360 overcomes this challenge by adding a Resubmitted tag to the resubmitted runs.

Manual resubmission

Manual resubmission of runs is a straightforward operation, select the runs to be resubmitted in the run history of the respective Logic App and click on Resubmit runs option. It is also possible to resubmit runs in bulk.

Automated resubmission

The manual resubmission is handy when the number of runs to be resubmitted is less. But automated resubmission is useful when there are many runs to be resubmitted. Automated resubmission has the below advanced features to enhance the efficiency of resubmission.

Option to include or exclude the already resubmitted runs.
Resubmitting runs based on one or more error reasons.
Resubmitting runs from a specific trigger.
Resubmitting runs with the selected run actions based on their state.

A more convenient way than manually running an automated task each time there is a violation to configure the automated task as part of the monitor rule that will be executed each time the configured rule is violated.

What if you need to do more

Serverless360 Business Applications are aimed at providing the tools your Support Operator needs to perform daily operations for your integration solutions.

You may want to allow less experienced support users or Business Super Users to have visibility of your integration processes and to be able to perform a level of self-service. In this case, Serverless360 provides a Business Activity Monitoring module that can be used alongside Business Applications, allowing you to provide an even more fantastic experience for your users.

Conclusion

Tracking Azure Logic Apps failures is essential to ensure the smooth functioning of the business. Azure provides Logic App monitoring via Azure Monitor by configuring alert rules. The alert rules help to detect failures and performance bottlenecks. But, monitoring multiple Logic Apps requires repeated alert configurations multiple times, making it challenging to track the alerts.

By combining Logic Apps into a logical container and using Monitoring Profiles to track the failures, Serverless360 increases its edge over Microsoft Azure. This helps to identify and mitigate the issues proactively, reducing downtime and increasing productivity. Having features like Automated resubmission in case of violations will be particularly useful.

Azure offers fundamental monitoring features and works well with a few Logic Apps. Serverless360 is the go-to option for businesses managing multiple resources.

Experience Business Application with a 15-day free trial!

Monitoring Azure Integration Services with Proactive Strategies

Sabari Rohith — Fri, 19 May 2023 11:53:14 +0000

Enterprises are increasingly turning to cloud-based integration solutions to streamline their application development and management processes. Azure Integration Services is a cloud-based integration platform provided by Microsoft, designed to facilitate the integration of various enterprise applications and systems. It offers a range of tools and services that help to simplify and accelerate the development of enterprise applications, as well as improve their scalability, reliability, and security. The below diagram shows some of the common resources on Azure which are used in the Azure Integration Services offering.

Effective monitoring is a crucial aspect of any Azure Integration Services implementation. Without proactive monitoring, it can be difficult to identify and address issues before they become critical. But creating a fail-proof monitoring strategy for Azure Integration Services can be challenging. In this blog post, we’ll explore the key components of a proactive monitoring strategy for Azure Integration Services that will help you stay on top of potential issues and ensure smooth operations for your business-critical applications.

Scenario: A booking management sequence

To understand the power of the Azure Integration Services platform, let us consider a booking management system that may have a flow similar to the below steps:

A customer initiates a booking request through a mobile app or website.

The request is sent to the business application, which processes it and validates the request based on various criteria such as availability, pricing, and other business rules.
The business application interacts with the existing ERP (Enterprise Resource Planning) system to ensure that the requested product or service is available and that it can be fulfilled within the specified timeframe.
The CRM (Customer Relationship Management) system is also consulted to ensure that the customer’s data and history are up-to-date and that any relevant preferences or special requests are considered.
If the request is approved, the business application generates an order or booking confirmation and sends it to the customer via the mobile app or website.
The ERP system updates its inventory and order management systems to reflect the new transaction.
The CRM system updates its customer database with new orders or booking information, including any feedback or ratings provided by the customer.
The business application generates any necessary notifications or alerts to internal stakeholders, such as fulfillment teams or support personnel, to ensure that the order or booking is processed and delivered as expected.

Logical Architecture

To implement a solution that would meet the needs of the business requirements we could implement a logical architecture shown in the below diagram. This would give us the capabilities needed to provide a lightweight and flexible solution making the most of modern cloud platforms.

Each block identified within the diagram has a specific purpose:

Messaging Service – The Customer facing website or mobile app needs to be decoupled from the backend Orchestrator to deal with any unpredictable spikes in traffic. This asynchronous messaging pattern also ensures durability to survive intermittent failures.
Event Routing Service– The goal is to eliminate polling and save the associated cost and latency. The orchestrator service should be informed of the availability of a new booking message in the messaging service.
Orchestrator Service – This is the heart of the business application. The entire workflow of processing the booking will be defined in this service. Integrating the existing Cloud or on-premises systems like ERP and CRM is necessary to complete the booking.
API Management Service – Connect on-premises systems like the ERP and CRM applications to the Orchestrator Cloud Service to safely integrate the Cloud and on-premises environments. Protect the APIs with keys, tokens, and IP filtering. As the orchestrator definition evolves, the existing client applications should continue functioning without modification.
Compute Service – The service should allow end-to-end development experience in a language the existing team has expertise in to solve complex orchestration requirements. The need is to build and deploy locally without additional setup, deploy and operate at scale in the Cloud, and integrate with other services using triggers and bindings.
Data Transformation Service– The data from the external ERP and CRM systems need to be transformed and stored in the SQL database for the computing service to use.

Designing an application using Azure Integration Services

With Azure, an Enterprise Integration application can be built by assembling Lego blocks. Choosing the appropriate Azure Integration Service is critical in building the Integration Application correctly.

The above logical architecture could be implemented using the following Azure Integration Service components:

Service Bus
Event Grid
Logic Apps
API Management
Azure Functions
Azure Data Factory

The implementation of the logical architecture would look something like the below diagram.

Let us understand the capabilities of each Azure Integration Service in the diagram above.

Service Bus – Azure Service Bus provides a reliable cloud messaging service between applications and services, even when they are offline. Moreover, the Enterprise must ensure order processing and/or duplicate detection can be achieved through simple configurations, saving considerable time and effort. In this scenario, as the requirement is 1-1 decoupling, Service Bus Queue can be the best fit. Enabling Sessions ensures ordered processing; duplicate detection can be turned on with a duplicate detection time frame window.
Event Grid – Manages the routing of all events from any source to the destination. In this case, event-driven architecture is adopted to prevent the Orchestrator service from polling the Service Bus queue for any new message. The Event Grid will trigger an event to the downstream application on the arrival of the new message, hence being cost-effective.
Logic App – Seamlessly and securely connect Logic Apps to cloud-based and on-premise solutions, in this case, to the ERP and CRM solutions taking advantage of the hundreds of out-of-the-box connectors available and implement complex workflows that orchestrate across your services
API Management (APIM) – Inbound policies in APIM help control how data and services are being exposed to participating applications by allowing the definition of authentication, authorization, and usage limits. We can meet the security and compliance requirements with unified management experience across all internal and external APIs.
Azure Functions – Functions extensions on Visual Studio and Visual Studio Code facilitate efficient development in the preferred language on a local machine, fully integrated with the Azure platform. Continuous Integration and Continuous Delivery (CI/CD) can be achieved using Azure pipeline definitions. With a programming model based on triggers and bindings, use triggers to define how functions should get invoked and use bindings to connect to other services declaratively.
Azure Data Factory – Offers a single solution to ingest data from diverse and multiple sources allowing support to 90 built-in connectors to acquire data from Big Data sources like Amazon Redshift, Google BigQuery, and HDFS; enterprise data warehouses like Oracle Exadata, and Teradata; SaaS apps like Salesforce, Marketo, and ServiceNow; and all Azure data services.

Azure Integration Services provides a microservices-friendly approach allowing you to build more scalable and stable event-driven applications!

The importance of monitoring Azure Integration Services

Integration is the heart of modern business applications. In any typical Enterprise, distributed subsystems are integrated to complete the business workflow. Failure in these integrations should be spotted and fixed before it impacts the customer. Hence, it is critical to enable continuous monitoring of the underlying integration.

Consider the business application mentioned above; what if the Logic App listening to the incoming messages to Service Bus Queue goes down?

Incoming messages will not be processed, they may end up stuck on the queue or errors processing then will end up with messages on the dead letter queue.
The Integration is broken!

What does it mean to the business?

Orders are blocked
Failure to serve the customer on time.
Loss of business and goodwill.

How to prevent this?

The entire integration should be continuously monitored for any failures.
Access the right toolset to analyze the root cause and fix the failure before business impacts.

Available monitoring solutions

The Azure portal offers native monitoring solutions to monitor Azure Integration Services on their metrics. Choosing the key performance indicators can help to keep an eye on any failure.

Monitoring the ‘Runs Failed’ and ‘Trigger Failed’ metrics on a Logic App can help detect failed runs.

Upon detecting a failure in the Logic App, one can investigate the Trigger and Run histories to gather more details on the failure.

Enabling Log Analytics and Application Insights can also help you to analyze the root cause of a failure.

However, we need to be aware of the challenges in operating the application in the real world:

The Azure portal lacks one all-inclusive view of the entire Azure estate with the real-time status of the participating Azure Integration Services. In most cases, the Azure Integration Services will spread across multiple Azure subscriptions, making the scenario even more complex. For efficient operations, it is necessary to spotlight the failure, which is challenging through the interface offered by the Azure portal.
Alert Storm – The scope of the Azure metric monitoring alert rule is restricted to a resource; hence to get the status of the entire integration, multiple alert rules should be created, leading to several alerts. Also, note that there is a cost associated with every alert configuration. After all the efforts and costs, ending up with many discrete alert reports is less valuable.
Azure Experts – Log Analytics and Application insights are excellent tools for debugging or performing a root cause analysis. However, they invite Azure experts to be involved in Azure support, which leads to the elevated total cost of ownership, one of the biggest challenges Enterprises are keen to solve. Shifting left the operations is the desired solution, but the Azure portal is too complex for the Operations team with less Azure expertise.

In our decade of experience working with Integration customers, we observed that 90% of the issues in production are functional or data errors. The challenge for most customers is that support of the applications can only be performed by developers and experts in the implementation of the solution. The support or operations team cannot easily be involved in the day-to-day operation of your solution because of the restrictive tooling and significant skill sets required.

The Serverless360 value proposition is to lower those barriers to supporting your solution and to make it easier for your operations team to take charge of the day-to-day operations and only need your developers and experts by exception rather than by default. We see this as shifting support to the left in your organization.

This Azure support strategy can save considerable time and effort for Azure Experts, allowing them to focus on innovation for business and ultimately reducing the Total Cost of Ownership.

How is Serverless360 different from any APM products?

Numerous Application Performance Monitoring (APM) products are available for monitoring Azure Integration Services. Any APM will only detect a problem in the Azure Integration, but what is more important is the deeply integrated toolset to fix the issue and restore the business. Serverless360 is crafted with capabilities to complement the Azure portal and aims at ‘Shift Left Support.’

Essential monitoring perspectives for Azure Integration Services

An Azure integration should be monitored from 4 essential perspectives:

Static monitoring of the status of the participating resources, key performance indicators from the available metrics, and resource properties.
Run time monitoring, real-time end-to-end tracking of the message flowing through the Azure Integration Services.
Keep an eye on the evolution of the Azure subscription and document the platform for governance and audit purposes
Cost monitoring to be aware of your efficiency and to prevent unexpected costs

Serverless360 offers four core modules, each mapping to one of the perspectives mentioned above ideal for different Stakeholders involved in Cloud Operations. Below, you will find an overview of those modules.

Unified Observability – Business Applications

An Azure subscription can grow much faster than we expect. Below is a view of resources in a subscription in the Azure portal.

The resources exist in siloes in the Azure portal. Moreover, one cannot derive critical information required to support integration in production, like ‘Which integration this resource belongs to?’ or ‘What is its current status?’

Serverless360 can offer an operation-friendly interface that allows visualizing the siloed resources in an application context, spotting the failure in service, and offering the toolset to fix the identified issue and fulfilling the security requirement of granting the need-only permissions to the authorized team members.

Below is the representation of the same set of resources as Business Applications. One can clearly understand which application in my Azure estate needs immediate attention. The best part is that the business applications setting can be done in an hour and does not involve much effort.

Business Applications in Serverless360 allow the definition of Auto Map rules. Configure a rule to map any resource with a specific tag or any resource created in a specific resource group to a Business Application. Monitoring profiles can be configured to monitor the key performance metrics on the Azure resource mapped to a Business Application. That’s it; Serverless360 will continuously monitor the resources as your Azure estate evolves. The Operations team can rely on the Serverless360 dashboard to detect issues in your entire Azure estate.

A resource mapped to a business application can be extensively monitored on its status, properties, and metrics. The Service Map feature can help the team visualize the resources as an application and indicate the resource status. The deeply integrated tooling for failure management enables the support team to restore failures like resubmitting the dead letters in Service Bus or Event Grid and reprocessing the failed executions in Logic Apps, Data Factory pipelines, and Azure Functions.

End-to-end tracking – Business activity monitoring

Business operations often get struck with the question, “Where is my message?” In a modern distributed application context, this is a much more complex question to be addressed.

Business users may not have a technical understanding of the application, but they will have a sound understanding of the functions and the data associated with the business.

Introducing Serverless360 BAM for business users unlocks achieving end-to-end tracking of messages flowing through the distributed application.

The best part here is that Serverless360 BAM is technology-agnostic and platform-independent. Enable your business users to locate a specific transaction, spot any exceptions or long-running unresponsive processes, or derive strategic business decisions.

Azure Data to Insights – Azure Documenter

Enterprise integrations in Azure evolve in no time. Even before we realize the volume of Azure resources grows massive, it becomes hard to derive insights from their usage.

Azure Documenter in Serverless360 offers reports to understand the evolution of the subscription(s).

Consider that every team is assigned a resource group to build their integration. The Resource Auditing report in Azure Documenter can compare the current month’s spending against the last month’s, an executive summary of the entire subscription.

Reporting the security compliance of the application is also made simple.

Cost control – Cost Analyzer

One of the key reasons organizations choose to migrate to the Cloud is to cut off the pre-computation cost on the infrastructure. While taking advantage of the pay-as-you-go model, monitoring Azure spending is crucial.

Cost Analyzer in Serverless360 allows cloud economists to create a single pane view to deliver the desired insights across subscriptions to drive actions.

Setting up budget alerts and optimization schedules to cut unnecessary costs are essential tools for cost control.

Business Value

With Serverless360, enterprises across several business domains have devised a friction-free support strategy to shift- left the support tasks in the support escalation path, allowing the Azure experts to focus on business innovation. The result is a significant reduction in the Total Cost of Ownership and considerable improvement in the resolution time, ensuring Customer Delight through operational excellence.

Conclusion

To sum it up, Serverless360 is an advanced cloud management platform that offers comprehensive monitoring of 4 essential perspectives to address the requirements of various stakeholders involved in Azure Operations. Serverless360 stands out in comparison with several APM solutions available in the market, as an APM can only help detect an issue, whereas Serverless360 is perfect to Visualize, Spot, and Fix issues with the underlying integration.

I strongly recommend checking out the Enterprise case studies on devising an Azure support strategy using Serverless360.

If you want to discuss how Serverless360 can help address your challenges with administering your Azure integrations, why not contact us? We are always happy to discuss any challenges over a call. If there is a match, we can give you an obligation-free demo of the product, or you can take a 15 days free-ride to explore the product yourself.

Azure Resource Monitoring: The key method for Holistic Monitoring

Sabari Rohith — Mon, 01 May 2023 10:09:55 +0000

Introduction

When your organization has started to adopt Azure, the continuously increasing number of Azure resources throughout all your Development, Test, and Production subscriptions make it hard to keep on top of the health of all those resources. So, it is important to have proactive Azure resource monitoring to know when something unexpected happens.

This blog looks at the different types of monitoring, manual monitoring difficulties, and the difference between Application Performance Monitoring products and Cloud Management products.

Different types of monitoring

Before looking into the other topics, let’s first discuss what kind of monitoring is relevant to be aware of the well-being of your Azure resources.

The following types should be considered for monitoring your Azure resources:

Availability monitoring
Health monitoring
Performance monitoring

Let’s spend some time exploring what those Azure Monitoring types are about.

Availability monitoring

All solution components must work as expected to meet the business requirements. KPIs are often used to monitor the overall availability of a solution. Such KPIs include Overall availability, Planned unavailability, and Unplanned availability.

The expectations around solution availability can be registered as Service Level Agreements (SLAs). In such contracts, you can formalize subjects like Solution availability (based on KPIs), Performance Metrics, Response times, Planned Maintenance, and Usage statistics.

Health monitoring

You also want to be aware of the health of the individual components of a solution, which concerns setting up the rules and conditions to determine the health of those components. Depending on the importance of the components, you can think of different notification policies.

For example, you can think of immediate notifications in case of critical components, delayed notifications in case of components of lesser importance, or scheduled health reports. You can also be aware of anomalies via health dashboards, scheduled health reports, etc.

Performance monitoring

It is also essential to keep an eye on the solution’s performance. Functionally, it might work flawlessly, but if the solution is under stress because it is under-resourced, it cannot effectively be used. To understand the solution from a performance perspective, you could monitor, for example, the number of processed transactions within a specific timeframe or the number of requests per second. Besides technical performance, you could also monitor user satisfaction using an Apdex score.

Manual versus Automated monitoring

Now that we understand the relevant types of monitoring, let’s also look briefly at how monitoring is taking place; manually or by using automated tooling.

Manual monitoring

When monitoring needs to be done manually, administrators or support engineers must frequently check the status of the components comprising a solution. Manual monitoring requires proper procedures for monitoring responsibilities, skilled and disciplined engineers who perform the monitoring tasks, and precise escalation levels. Consider setting up proper dashboards in the Azure portal to reduce the daily monitoring effort.

Automated monitoring

Monitoring can also be automated, and you can use anything from a scheduled PowerShell script to Azure out-of-the-box features or a third-party solution. Each choice comes with its own set of pros and cons. Script-based monitoring will be free but cumbersome and limited in capabilities. Third-party monitoring solutions, however, can be compelling but, in most cases, require license fees and a learning curve to get familiarized with the product.

Challenges with manual monitoring

If your organization still needs to (yet) have a monitoring solution, monitoring has to be done manually. However, with that, several challenges arise. Such challenges include:

Consistently performing the monitoring over time.
Continuity of monitoring during any leave.
The required skill level of monitoring engineers.
Passive rather than proactive monitoring.

If not appropriately addressed, those challenges will bring the well-being of your Azure solutions at stake, so it is worth exploring automated monitoring tooling.

Application Performance Monitoring solution or Cloud Management solution?

When you have decided to investigate purchasing a monitoring solution, there is still a choice to be made. Will you go for an Application Performance Monitoring (APM) or Cloud Management solution? Let’s look at a couple of pros and cons of both categories.

Focus versus comprehensiveness

APM tools are focused on monitoring and optimizing application performance. On the other side, Cloud Management tools have a broader scope; besides monitoring capabilities, such products can also provide operational capabilities. This enables administrators to manage all aspects of their Azure solutions under one hood, which might be lacking with APM products.

Customizability versus generic

By nature, APM products are focused on application performance. Because of that, such products provide a high degree of customizability regarding what can be monitored. With Cloud Management platforms, monitoring is more generic, and you may be unable to monitor the same metrics as with APM products.

Additional costs versus cost-effectiveness

Since APM products focus solely on monitoring application performance, additional tooling might be required to monitor other aspects or administrative purposes. This would add additional costs in terms of licensing, etc. Cloud Management platforms might be a better choice because such products provide both monitoring and administrative capabilities.

Cloud management with Serverless360

Serverless360 is a cloud management solution that enables you to keep control of your cloud solutions. To support different requirements, the product has four main modules. Those modules are:

Business Applications – To support your Azure solutions from a technical perspective.
Business Activity Monitoring (BAM) – To provide Business Users insight into their business transactions via portals and monitoring.
Azure Documenter – Generate ad-hoc and scheduled reports about your Azure resources.
Cost Analyzer – Get a grip on your Azure spending with cross-subscription overviews, budget monitoring, and resource optimization.

Most of those modules have different kinds of monitoring, but in this article, we mainly focus on Business Applications, so let’s have a little bit better look at that module in Serverless360.

With Business Applications, you can bundle the Azure resources across multiple Azure subscriptions, Resource Groups, and Tags that belong to an Azure solution into Business Application containers. Access to those Azure solutions is provided at the level of those business Applications. Within those Business Applications, you will find comprehensive monitoring and administrative features.

The monitoring capabilities have rich means to send notifications of anomalies and contain automated discovery, thereby supporting the availability of your solution. Within Business Applications, you can also automate maintenance tasks like cleaning up dead-lettered messages, resubmitting Logic App-runs, etc.

Automated monitoring with Serverless360

As mentioned earlier in this article, with most modules of Serverless360, you can perform automated monitoring. Those modules are:

Business Applications – Availability, Health, and Performance Monitoring.
Business Activity Monitoring – Business Transaction Monitoring.
Cost Analyzer – Actual/Amortized Costs Monitoring.

Let’s now focus on several kinds of automated monitoring that can be done with Business Applications.

Availability Monitoring for Business Applications

In the Business Applications module, you group components that belong to the same solution into logical containers, aka Business applications. To understand the overall availability, automated monitoring can be done against the status of all the Resources. When resources are not in their expected state, you will receive notifications. You can configure parameters like the monitoring frequency and when you want to receive the notifications.

Monitoring an API Endpoint

Consider an important API Endpoint you might have that you want to bring under monitoring. For example, you can think of monitoring the endpoint against a specific HTTP return code, like HTTP equals 200. If the API Endpoint does not return an HTTP 200, then you want a ticket to be created in your ServiceNow ticketing system, enabling the appropriate team to investigate the issue.

Scenarios like this can easily be set up with Serverless360. Actually, the monitoring and notification capabilities go further than just that; with respect to API Endpoints, you can monitor:

HTTP return codes
API Endpoint Response times
Validate Text/XML/JSON responses

Health Monitoring for Business Applications

For Health Monitoring purposes within Business Applications, you can schedule health reports that provide insights into the well-being of the resources within those Business Applications. You can configure how often you want to receive those reports and if you want to receive them only in case of anomalies.

Monitoring failed Logic App runs

Besides monitoring the state of Resources, with Serverless360, you can also monitor the health of, for example, Logic App runs. This enables you to be aware of failed Logic App runs immediately after they happen, and take adequate actions. With the Automated Tasks feature in Serverless360, it is even possible to create a task to automatically resubmit failed runs. This is not only improving the overall health of your solutions but also reduces your workload!

Receive notifications where you need them most

Serverless360 Business Applications are flexible in terms of notification channels. You can receive notifications via:

Collaboration platforms Microsoft Teams and Slack
Ticketing systems ServiceNow and OpsGenie
Azure DevOps, Azure OMS
Email, PagerDuty, Twilio
Webhook

By choosing the right notification channel, the right people will be informed, and notifications won’t go unseen.

Conclusion

When you are looking for a cost-effective, ever-evolving solution to manage your Azure resources, Serverless360 might be the right product for you. You can read more about the product on the Serverless360 website. There, we provide an obligation-free trial, but you could also discuss your challenges regarding managing your Azure resources with our team over a demo. They are happy to go on a call and see if Serverless360 would fit your organization and show you the product in detail.

10 Ways to Optimize your Azure cost

Sabari Rohith — Mon, 24 Apr 2023 04:08:01 +0000

``In modern times, building and publishing an application has become very easy with Cloud-based deployment. Users don’t need to worry about infrastructure-related challenges like availability, reliability, scalability, etc. The cloud providers are responsible for keeping the deployment flow simple and intact.

Providing many advantages and coherence, the high cost incurred for such benefits is the downside. If the resources created in Cloud are not adequately audited or kept track of, it will quickly bring up the cost to double or triple the allocated budget. Some cases even drained a year’s tech budget in a month.

This article will help you understand some of the Azure Cost Optimization best practices.

1. Understanding the Resources by Tagging and Grouping

Azure Resource Groups are containers that hold related resources for Azure solutions. Creating resources under proper and more petite Resource Groups will help the users understand the purpose of the resources, which also helps in cleaning up the resources related to a new solution.

Many users need help creating proper Resource Groups before adding resources. As this is a one-time process, you can only do a little with the existing resources that are not grouped during the creation. For such scenarios, Azure Tags will be the right solution. Here is a guide from Microsoft for naming and tagging resources.

Once the resources are correctly grouped, it is easy to understand how the cost is distributed across the solutions. Visualizing and getting insights based on filters like Resource Groups, Types, Regions, and Tags are available at each Subscription level in the Azure Portal. For a better experience at the scope of multiple Subscriptions, try out Serverless360 Cost Analyzer, which is available for a 15-day trial.

2. Monitor the Cost and Act on Unintentional Spikes

Once you understand how the resources are distributed across multiple solutions and the cost it incurs, the next step is understanding the daily spent rate of your Azure subscriptions.

You can calculate the daily spent rate by taking the average day-wise cost for the last 7 days. Cost spikes due to yearly renewals like App Service certificates should be excluded so that you can arrive at the exact rate.

You can create budgets in the Azure portal or other third-party tools at a Subscription level. It would help if you had tools that support monitoring the cost at multiple Subscription levels. Serverless360 calculates the daily spent rate automatically while configuring cost monitoring.

Once the monitor is set up, you will be alerted whenever something goes off the chart. You can immediately check which resource contributed to the raise and act on it. If the raise is valid and the resources are created or scaled intentionally, you must calculate the daily spend rate again and monitor it. There are tools available to alert if the cost trend changes automatically.

3. Smart start/ stop or Deallocate Nonproduction Resources

Close to 70% of a month, the resources contributing to the nonproduction environments will run for no reason. Most resources can be stopped, scaled down, or deallocated during off-business hours or weekends, which can be achieved using 3rd party tools like Serverless360.

4. Identify and Remove the Idle Resources

While grouping the resources using Resource Groups and Tags, you could have identified the resources that serve no purpose and can be deleted immediately. For example, most of the time, disks associated with the VMs will be left idle while deleting the VMs; you can delete those disks.

You can find the unidentified idle resources by analyzing the consumption metrics like CPU and Memory percentage. You can remove the resources with zero consumption.

5. Reservations

Reservation is one big area where you can save up to 80% on the cost spent on resources. Microsoft Azure discounts resources based on usage commitment declared for 1 to 3 years. Reservation is not just about discounts; you can get a high-performing machine at a shallow price point. You can find the details about the reservations here.

6. Savings Plan

Savings plans are, just like reservations, a recent addition. You need to commit to the fixed hourly spent for 1 to 3 years and get up to 65% discount on it.

For Reservations, you need to select the VM sizes upfront for 3 years, which might be challenging for growing businesses as changing the size of the VMs in reservations comes with a hard limit of only up to $50K per year. Saving plans expect the hourly cost to be committed, and you can create VMs of any size within the committed limit.

The Savings Plan is supported for computing resources like Reservations. You can find the details about the Savings plans here.

7. Spot Instances

Spot instances are another offering by Microsoft Azure, where you can purchase the unused capacity of the VMs at a significant discount. These VMs are suggested for interruptible workloads, and you can get up to a 90% discount.

You can use Spot instances for Azure DevOps pipeline build agents, interruptible batch jobs, etc. You can find the details about the spot instances here.

8. Right-sizing

Provisioning servers in Azure is effortless. Within a few clicks, you can create a machine costing 1000s of dollars, leaving the engineering users to provision more than the required capacity to be on the safer side.

Though the environments will be stable and reliable by hosting in a giant machine, the cost comes as the downside. You can easily consume a significant chunk of the budget allocated by provisioning a few servers. It is always recommended to evaluate resource utilization and right-size them continuously. Right-sizing comes as a feature in many 3rd party tools.

9. Choosing the Right Services

There are many Serverless resources available in Azure. Instead of paying for the total capacity of a server, we can pay just for the capacity we use. Most of the time, it is better to go with Serverless resources as it automatically saves the cost during the system’s idle time.

Azure Functions running in a consumption plan are a suitable replacement for many background tasks running in resources like VMs, and Cloud Service Web/Worker roles.

Similarly, many other services come in a Serverless plan. Consider using Serverless resources in the possible areas to save cost.

10. Setup Policies and Follow Best Practices

It is common to have technical brainstorming sessions, handover meetings, and retrospection meetings for engineering-related activities. Plan the same kind of meeting regularly to discuss cloud cost optimization.

It would help if you defined Access policies to restrict resource creation and modification, and they should be delegated to team owners. Only some team members should be allowed to deploy resources as it comes with a cost.

For obtaining the best overview, you should subscribe to a 3rd party tool for managing the Azure cost. It is as essential as an APM tool for an application, thereby ensuring that all the Azure cost optimization best practices are followed.

Key Takeaways from Azure Cost Optimization Best Practices

Following the above 10 best practices immensely helps to optimize the Azure cost and have it under full control. But, in real-time, there is this challenge where the person who wants to understand the costs might not have the knowledge to break down the cost data into something meaningful, and the application teams do not have the access or tools to analyze their Azure spend.

Here is where Serverless360 Cost Analyser enables organizations to tackle this problem from both sides. The application teams can build cost analysis, budgeting, and optimization focused on making sure they are efficiently running their resources. Then they can work with the cost owner to build views which allows them to demonstrate they are being efficient.

This creates an all-around better experience where the application teams can be responsible for their costs, and the cost owner can focus on governance across teams. Cost Analyzer will help break up those silos which lead to zero transparency, inefficiency, and out-of-control costs.

Optimize Azure cost now for free!