DEV Community: Modhana

Explore Azure costs for multiple subscriptions with cost analysis

Modhana — Sun, 18 Dec 2022 15:09:35 +0000

In the fast-growing Azure space, it is essential to scale your business as Azure scales up. Azure is cost efficient by providing a pay-as-you-go model, but it is still necessary for enterprise users to undergo a Cost Analysis to keep their budget at stack. Let us consider a scenario, you’re a manager in your organization, and your team has been using Azure for the last several months. It has created multiple resources that cost money. The need for you is to understand Cost Management best practices and want to analyze which resources are the primary drivers of your monthly bill. Also, you would like to set thresholds on your team’s Azure expenses to stay within a budget.

For such scenarios, Azure provides a native solution, Azure Cost Analysis. The Azure Cost Analysis is a free tool that lets you see your Azure cloud spending habits. This data is crucial for effectively controlling and optimizing Azure expenditures.

In this blog, we’ll be focusing on,

How do you do cost analysis in Azure?
Grouping and filtering options in Azure Cost Analysis
Understand billing and resource management scopes
Use cost alerts to monitor usage and spending
Monitoring multiple Azure subscriptions with Serverless360

How do you do cost analysis in Azure?

Cost analysis in Azure can be performed by selecting a subscription from the Subscriptions section and navigating to cost analysis under Cost Management tools.

After the analysis, you can view aggregated expenses, determine where expenditures are incurred, and spot trends in Azure spending. Additionally, you can track costs accumulated over time and use this knowledge to predict quarterly, monthly, or annual cost patterns compared to a specific budget.

Microsoft Azure only supports cost monitoring and visualization for a single subscription.

Can we have multiple subscriptions in Azure?

Multiple subscriptions with different access management policies and billing procedures can be added to an Azure account. The following boundaries can be defined using an Azure subscription:

Billing boundary: The billing requirements for utilizing resources are defined by this subscription type. You can create separate subscriptions for different billing needs, and Azure will send independent billing resources for each subscription.

Access control boundary: To represent different organizational structures, you can create an access control boundary at the subscription level by applying access management policies to additional subscriptions.

Why is it recommended to use multiple Azure subscriptions?
Some resource types are subject to subscription limits. When a subscription reaches these limits, you must create other subscriptions and allocate resources for further business orchestrations.

Multiple Azure subscriptions are frequently used by organizations to avoid per-subscription resource limits and to manage better and govern their Azure resources. It is critical to developing a subscription scaling strategy.

You can create a management group hierarchy to manage multiple Azure subscriptions using an Azure account to manage all your subscriptions and resources.

Which resource groups cost the most so far this month?

Azure Cost Analysis view allows you to visualize cost usage by applying filters over a specified time. You can determine the maximum cost incurred by a resource group (or) resource for any customized date range.

You can determine which resource groups are used most expensively for a particular month by choosing the resource group option from the list of preview views while defining a cost analysis view and specifying the time interval.

Which resources cost the most so far this month? Are there any subscription cost anomalies?

You can determine which resource is used most expensively for a particular month by selecting the resource option from the list of preview views while defining a cost analysis view and specifying the time interval.

Anomaly detection in Cost Analysis helps you have prior knowledge of cost fluctuation over a period without any surprises. It allows you to understand if the cost usage pattern is unusual.

Navigate to Cost Management from Azure Home in the Azure portal. Choose a subscription scope, then select Cost analysis from the left menu. Select any view from the Preview views section of the view list. If available, you’ll notice the anomalies as a part of insights if you have any cost anomalies.

How many reservations are being used? Which resources are utilizing reservations?

Reservations offer a billing discount while not affecting your resources’ runtime state. When you make a reservation, the discount is applied automatically to matching resources.

You can purchase Azure Reservations by committing to one-year or three-year plans for,

Virtual Machines
Azure Blob storage or Azure Data Lake Storage Gen2
SQL Database compute capacity
Azure Cosmos DB throughput and other Azure resources

Only resources covered by an Enterprise Agreement, Microsoft Customer Agreement, Cloud Solution Provider (CSP), or subscriptions with pay-as-you-go rates qualify for a reservation discount.

All the purchased reservation utilizations can be viewed by navigating to Reservations in the Azure portal. To view the utilization history and specifics, select the utilization percentage.

Grouping and filtering options in Azure Cost Analysis
Azure Cost Analysis Views can be visualized by filtering your cost data based on shared properties such as meter category, subscriptions, resource groups, resource types, tags, etc.

You can organize your cost data by common properties such as meter category, resource group name, subscription, and tags.

Understand billing and resource management scopes

Billing is managing the business relationship and issuing customer invoices for goods or services. You can control all your payments, invoices, and cost tracking through your billing account.

Azure offers three resource management scopes. Each scope supports access and governance management, which may or may not include cost management.

Management groups are hierarchical containers used to arrange Azure subscriptions. A management group tree can support up to six levels of depth. The Root level and the subscription level are not included in the limit.
Subscriptions are the primary storage containers for Azure resources.
Resource groups are logical collections of connected resources for an Azure solution with a standard lifecycle. An illustration would be resources that are deployed and removed simultaneously.

Use cost alerts to monitor usage and spending
Azure Cost Analysis supports three alerts: budget, credit, and department spending quota.

The amount specified in the alert condition of the budget is what budget alerts let you know when your spending, based on usage or cost, reaches or exceeds.
Credit alerts inform you when your Azure Prepayment (previously called a financial commitment) has run out. Credit warnings are automatically generated at 90% and 100% of your Azure Prepayment credit balance.
When your department’s spending goes over a specific quota threshold, you’ll receive a notification from department spending quota alerts.

Though Azure Cost Analysis has its pros, it also has its cons. The feature set provided by Azure is restricted to a single subscription level. This restriction does not favor enterprise organizations as they usually work in multiple environments. Visualizing multiple environments under one roof would be a beneficial option for such an enterprise user. Moving further in the blog, we will discuss how such visualizations can be achieved with Serverless360.

Monitoring multiple Azure subscriptions with Serverless360
Serverless360’s Cost Analyzer is an all-in-one integrated solution that offers a variety of features to meet the user’s needs and requirements.

Analysis Views in Cost Analyzer can visualize multiple Azure subscription and resource costs simultaneously. Grouping and filtering options are provided to customize views according to the user’s preferences.

Monitors in Cost Analyzer allow you to set a monthly or daily budget for their Azure costs and receive notifications if the actual cost exceeds the budget. You can also have a preview of Azure cost trends to have prior knowledge before creating a monitor. Monitor rules for a single monitor can be specified across multiple subscriptions.

Cost Optimization in Cost Analyzer enables you to create optimization schedules with resources associated by defining the tiers, throughput values, and the resource state concerning the Up and Down hours of a week.

Summary
Serverless360’s Cost Analyzer is one of the best solutions that provide cost visualization and cost monitoring support for multiple Azure subscriptions.

Check out Cost Analyzer features by trying out the preview version currently available in the Serverless360 application for all licensing types.

Download Azure Service Bus messages using Serverless360

Modhana — Sun, 18 Dec 2022 14:51:00 +0000

From our experience handling Azure Service Bus messages, one frequent suggestion we get from the support person is to download messages from Azure Service Bus entities like Queues and Topic Subscriptions. By downloading the messages as a local copy, it becomes easy for them to debug the messages and use it at a later point in time. Basic knowledge of Azure Service Bus messaging entities is a pre-requisite for the better understanding of this feature.

Azure Service Bus Messages

Microsoft Azure Service Bus is used for delivering information in a reliable and a much easier way. The messaging service ensures that the information is delivered even if the two parties are never both online at the same time, or if they aren’t available at the exact same time.

Messages are sent by the sender to the receiver through the Azure Service Bus. The message transferred through Azure Service Bus consists of the actual message body and the properties of that message. The message may be sent to or received from the Service Bus Queues or Topic Subscriptions.

Message Processing Challenges in Azure Portal solved in Serverless360

Azure Portal does not facilitate viewing Azure Service Bus messages. But real-time business scenario would require processing these messages for various business scenarios. Serverless360 understands the requirement and comes up with the following message processing capabilities on Azure Service Bus Queues and Topics.

Send messages
Resubmit & Delete Messages
Resubmit & Delete Dead-Letter messages
Repair and Resubmit messages

The latest addition to the above capabilities is the feature to download messages. This blog focuses on the message download capability.

Business scenarios require Azure Service Bus messages to be downloaded for further processing. On demand from customers for saving messages for various purposes, Serverless360 has now got the capability to download messages as single or in bulk.

The messages will be downloaded as a zip file with text files containing the JSON object of the message. Each message will be saved in a separate text file.

Downloaded Message Content
The downloaded Message file holds the JSON content, that includes the following properties of the message:

Message Body
sequence number
message Id
size
schedule enqueued time
enqueued time
time to live
content type
system properties
the custom properties

File Format
The name of the zip folder will be the name of the Queue or Topic Subscription to which the messages belong. The name of the text file will be the Message-Id of that message. The file will have all the required properties of the message along with its content, which can be utilized for further processing

Azure Service Bus Dead-Letter Queue Monitoring

Modhana — Mon, 21 Nov 2022 09:09:35 +0000

Azure Service Bus dead-letter Queue

Azure Service Bus queues and topic subscriptions provide a secondary sub-queue, called a dead-letter queue. The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or messages that could not be processed. So, any message that resides in the dead-letter queue is called a dead-lettered message.

Dead-Letter Queue Reasons

Before investigating the best way to monitor and reprocess Service Bus messages efficiently, let us quickly hop in and understand the reasons behind why an active message get dead-lettered.

Most of the time we could notice that the message fails to reach the receivers due to the following reasons;

Dependent service not available
Faulty message
Process code issue

Dependent service not available

This could be one of the foremost and time after time reasons where the services that reply on message delivery may go down for a short period. For instance, Redis or SQL connection issues may often happen.

Faulty Message

If in case the message doesn’t have a mandatory parameter or some value is incorrect, then the message will end up in the dead-letter queue after the max delivery count is attained.

The failed delivery can also be caused by a few other reasons such as network failures, a deleted queue, a full queue, authentication failure, or a failure to deliver on time.

Here we can drill down the reasons into two ways:

System-level dead-lettering
Application-level dead-lettering

Reasons for System-level dead-lettering

Header Size Exceeded – Maximum header size: 64 KB
Error on processing subscription rule
Exceeding time to live value
Exceeding maxDeliveryCount – When DeliveryCount exceeds MaxDeliveryCount, the message is moved to the DLQ, specifying the ‘MaxDeliveryCountExceeded’ reason code. This behaviour cannot be disabled, but you can set MaxDeliveryCount to a very large number. You could specify the Maximum Delivery Count between 1 and 2147483647.
When Session id property is set to true (the default is false)

Reasons for Application-level dead-lettering

Messages that cannot be properly processed due to any sort of system issue
Messages that hold malformed payloads
Messages that fail authentication when some message-level security scheme is used

Monitoring Service Bus dead-lettered messages made easy

Be it Azure Service Bus Queue or Topic Subscriptions, Serverless360 makes it handy to monitor the dead-letter messages in your testing, pre-production and production environments.

It completely eliminates the complexity or limitation of Azure Monitor which might not fulfill your production environments requirements. For instance,

Monitor multiple Queues and Topics Subscriptions dead-lettered count in a single report
Monitor a Queue metrics against another metric, for instance, compare active messages vs dead-lettered messages
Consolidated monitoring of both the property values and status of Service Bus

I will take you through Serverless360 and showcase how quickly you can configure the alert rule to monitor the dead-lettered messages.

Once you signed up for a free trial, associate the Azure Service Bus in Business Application (Serverless360 feature) – this module lets you access, monitor and manage various Azure resources.

To configure rules, head to the desired resource type in the Manage resources section inside the Business Application. Select the respective queue or topic for which the rules should be configured, and choose the dead-letter count property with warning threshold and error threshold. That’s it.

Azure Service Bus dead letter queue resubmit

Serverless360 makes your life easier when it comes to repairing and resubmitting the dead-letter messages of queue or topic subscription providing out of the box features like

Advanced search option to find the dead-lettered message
Intelligent automation to bulk resubmit the dead-lettered messages
Variety of message receval modes like Peek-lock and Deffered mode
Back up and purge messages
Download particular set of messages

Now, look at the step-by-step process to resubmit the dead-lettered messages in Serverless360

Click the queue whose dead letter messages must be processed, this will lead to a message retrieval dashboard.
Select dead letter tab and click to get a specified count of messages

Select those messages to be resubmitted and click resubmit

Select the destination Queue and proceed to resubmit

The messages can also be repaired and resubmitted

The user can also view the properties, content and dead letter reason of a message

Conclusion

I hope this blog gave a better understanding of Azure Service Bus Dead Letter Queue, the reason behind a message getting dead lettered. We also discussed on how Serverless360 can help to process Dead letter messages much smarter way.

Distributed tracing and correlation through Service Bus messaging

Modhana — Thu, 10 Nov 2022 01:39:07 +0000

Over the last few years, Microsoft has built excellent tooling around different technologies. Today, everything is available and achievable through the Azure Portal, which helps manage complex solutions. However, this brings in challenges when it comes to managing the distributed resources. The struggle to keep track of messages or know the flow of messages through these distributed resources is growing day by day.

In most Integration services, Service Bus acts as a message broker to transfer data from a source to a destination.

Azure natively supports distributed tracing using Application Insights agents or SDKs for .NET, .NET Core, Java, Node.js, and JavaScript. When a Service Bus receives a message, it receives the data processed earlier by another system or scope. After receiving the messages from the Service Bus, the subscriber also performs a set of actions on the received data. To correlate these events of publisher and subscriber, there should be a unique property in the message which can be leveraged as a trace property for the End-to-End tracing.

Microsoft Azure Service Bus messaging has defined payload properties that producers and consumers should use to pass such trace context. The protocol is based on the HTTP Correlation protocol. Azure Service Bus SDK has a property that we can leverage to trace the messages.

Diagnostic-Id – Unique identifier of an external call from source to the queue.

Application Insights

Application Insights is a native offering from Azure, which helps to correlate events. These events can occur within the same application process or across several application processes
How Application Insights works is you insert a small package to your application and set up the Application Insights resource within Azure, thus sending the data to Azure to collect information. The business process is monitored, and it sends telemetry data to the Insight portal.

Querying the Distributed Traces

After a comprehensive design of the tracing approach and its implementation, we can now query our distributed traces in a meaningful way.

Once the Service Bus published the message and has logged tracing events, logs in Application Insights can be queried using the Kusto query language.

Application Map from Application Insights

Application Map helps you spot performance bottlenecks or failure hotspots across all components of your distributed application. Each node on the map represents an application component or its dependencies. If your app uses Azure services, you can click Azure diagnostics, such as SQL Database Advisor recommendations.

If you have a service bus pushing messages to a Function App or Logic App, this dependency can be viewed with the help of an Application map. This Application Map helps us understand how a service bus is a connecter to another resource.

Here is a sample of how the connectivity will be shown to a user.

You can use all of this Azure service to trace messages in a service bus. But the problem here is you need to reply or maintain multiple places. We don’t have a single place where we can see the entire flow of messages. When we consider enterprise integrations, their footprint will not be in Azure alone. They will have their data coming from an On-prem system or end up in an on-prem system. How do we correlate these hybrid integrations and visualize them in a single place?

Well, Serverless360 has a feature named Business Activity Monitoring (BAM) to track the flow of messages across your Business Process. You can create a flow to represent your business process and instrument it to trace the message. Once the instrumentation is done, all the data your business process tracks will be automatically displayed in Serverless360.

End to End tracking in BAM

Bring maximum visibility of your integration solution to functional support teams by defining properties to track business values at run-time. Easily correlate the flow of data within your system. Moreover, it allows you to visualize your business transactions in a graphical representation and detailed information. Furthermore, it also enables the visualization of a specific and more complex business process on how all milestones/transactions are executed in a particular sequence.

Associate Business Processes with Serverless360 to Track data

To start tracking your messages, you need to create a business process in BAM and picturise the flow your business process possess. This business process helps Serverless360 to understand how your Business Process has been designed. You can also have multiple Business processes, and each business process can have various transactions.

Instrument your Business process for Tracking

Once the flow has been created, the next step is to instrument your business process to make it suitable for Tracking. If you have completed your integration using Logic apps, we can leverage the custom connector facility from Azure to track the messages. If you have other components like Custom Apps, Function Apps, APIM, we can leverage the .Net library and SDKs from Serverless360.

Visualize the flow

When the instrumentation is done, Serverless360 tracks all the data your business process is processing. You can visualize the flow in which the message has travelled. Furthermore, it also provides the possibility to visualize a specific and more complex business process on how all milestones/transactions are executed in a particular sequence.

Spot failures in your Business Process

Spot the failures in your business process and know where the failure has occurred in your distributed components.

Search for Narrowed results

Search your transactions with some unique property in your message to spotlight that specific transaction.

Investigate the Messages processed

At any point in time, if you want to know the exact content that your business process has processed for a transaction, you can view it from the tracked window. Download any archived message directly from the Serverless360 portal at ease.

Reprocess a transaction

Have a failed transaction? You can retry single or multiple transactions right from Serverless360. Allow your support people to investigate the failure and reprocess it themselves without depending upon your skilled Azure resource.

Conclusion

Be it Native Azure Tracing features or out-of-box tracking capabilities by Serverless360, it is always to keep track of messages in our business. This Tracking helps us to understand the performance of a business and help to reduce the complexity.

Azure Documentation tool – the right way to monitor Azure usage

Modhana — Fri, 04 Nov 2022 04:52:38 +0000

As the adoption to cloud expand, subscribing and provisioning resources increases day by day. Enterprises have multiple strategies to implement their cloud deployments: single cloud, private cloud, multi-cloud, or hybrid cloud. There is a need to analyse and assess on regular basis to keep a watch on the usage to ensure expenditure on the cloud resources are efficient and there is no wastage.

There are various possibilities but will need some level of technical competence and regular maintenance:

Use Azure portal capabilities to configure asset management and monitoring
Build custom tools using APIs exposed by Microsoft Azure support
Use 3rd party tools and services

Not all stake holders will have access and the necessity to check through these tools, but it will be easy for the decision-makers to investigate the reports, summaries, comparison assessments, and useful charts or diagrammatic representations. This will help them to assess the usage briefly and take strategic decisions.

A well designed and articulated documentation of your Microsoft Azure environment can help in comprehending the usage easily without accessing too many tools and configurations.

Azure Documentation Generator
in Serverless360 helps you to document your Microsoft Azure Subscription. It comes with the capability to aggregate data from disparate resource providers in Microsoft Azure in a single report.

This allows for the creation of comprehensive technical documentation across resources enabling richer insights that would otherwise be impossible. Interpreting your cost and resource information on Microsoft Azure subscription into legible documentation is what Azure Documenter is for.

Microsoft Azure usage is constantly evolving

Microsoft Azure like any other cloud service provider makes it easy to instantiate and deploy resources at ease on a subscription basis. Though limits on usage and restricted user permissions could be set, the resource deployment and usage grow day by day due to the nature of the business demands for various purposes.

Resources could be deployed manually or through automated processes
Resources are deployed across geographical locations
Resources and environments are managed by different teams Configurations are set or automated to support different workloads/demands
Resources are billed across subscriptions and business units There is a need for different environments like Development, QA, Staging, Production, etc

Isn’t it necessary to have utmost clarity on Resource usage and their deployments, type, Configuration, SKU of these resources?

Deployment of resources in Microsoft Azure is made so simple that, without proper auditing and governance, this may lead to situations where resource management becomes uncontrollable.

There are certain aspects to the documentation:

What is to be documented?
How do we document?
Relating the documentation to the deployment and usage?
Keeping the documentation up to date?
Making it transparent and accessible to the relevant stakeholders?

Save time by using our Azure documentation tool

The solution is to use a documentation generator tool like Serverless360 Azure Documenter that helps to document the Azure resources that are running, regardless of how/why they were provisioned, the Azure region and subscription in which they are located.

We have made the configuration simple to get started with generating the documentation with few steps.

There is no separate installation or configuration change required in your Azure subscriptions. Serverless360 Azure Documenter requires Service Principal identity with ‘Reader’ permission to your Azure subscription. After you have configured your account, you can easily generate documentation of your cloud environment.

Our Azure documentation tool will automatically extract information about all the resources in your Microsoft Azure subscription through standard APIs and publish them.

Depending upon the size of your Azure deployment, the time taken for generating the document may differ.

What are the benefits of using Serverless360 Azure Documenter?

Azure Documenter saves a lot of time by automating up-to-date documentation of your Microsoft Azure environments. It generates a report on your assets and reduces manual effort.

Being able to easily document your Azure infrastructure will be most valuable to understand the usage.

It will provide stakeholders at all levels, confidence, and clarity of the adoption to Microsoft Azure cloud.*
It aggregates inventory of all your Resource groups. It eliminates manual activity to search and collect data about different resources and deployments.
It eliminates the need for a dedicated documentation team will save you a lot of time considering how little it costs.

Capabilities of Azure Documenter

You would be spending days creating and updating documentation but now you can spend time doing other important activities.

Serverless360 Azure Documenter provides capabilities to generate:

Executive Summary

Key decision-makers may not need all the intricate details of the cloud implementation and technical aspects. All they need is a summary at a high level:

What is the usage?
Where are the wastages?
Are the compliance and security guidelines met?
What is the cost associated?
Advisory and guidelines

Billing & Metrics

Generate documentation with vital information like billing and metrics for the overall/group-wise resources in the subscription. Provides a graphical representation of cost incurred resource wise, resource type-wise, location-wise and, resource group-wise. It also provides a split-up of cost consumed at the individual resource level.

Change tracking reports

As the business grows, Azure cloud implementations could expand vastly with thousands of resources and groups cutting across multiple subscriptions and environments. Due to changes in architecture, infrastructure, business logic, upgrades, and migrations, there will be a lot of changes in the resource configuration, addition, and removal, etc on a day-to-day basis. Not all these are done manually, most of them are directed through DevOps and are automated. Also, there are number of teams and people involved.

Keeping a track of all these changes, interpreting the need, etc becomes very important. This will help to understand and check if there were any potential modifications that will impact the application and eventually the business.

Azure Security Compliance reports

Primary need for any infrastructure deployment is to ensure security and compliance of data, practices, access, and use. Adopting to Microsoft Azure is no different. From time to time, it is important to have absolute knowledge and control of data security, data losses due to storage, migration and archives, any security breaches.

These reports provide tracking of implementation of compliance rules, following organisation standards and practices, and for auditing purposes.

Azure cost analysis

Businesses across the world are embracing cloud computing than ever before. They look at migrations, new implementations, and hence the interest towards cloud implementations is only increasing day by day.

“Our commercial cloud revenue grew 36% year over year to $19.5 billion.” said Microsoft Corp. announced the results for the quarter ended June 30, 2021.

This means a lot of enterprises are spending more on the cloud. There could be an overspend if there are no periodic assessments on the expenses. Though cloud vendors promote pay-as-you-go models and various other cost control practices, in reality, the expenditure pattern is different. There are platform costs, most of which are fixed based on the configuration, some cost spiral up exponentially based on usage unless controlled, there are hidden costs as well.

You may also want to compare and analyse cost between two different months.

Schedule documentation and notify stake holders

Generating documentation for an established and evolving Azure estate with thousands of resources may take so long due to data collection, computation, and analysis. It is not practical to manually initiate documentation at the intended time and for many subscriptions and configurations. Relevant stakeholders may have to be notified once the documentation is complete. Azure Documenter provides automation to address these requirements. Notifications are sent to widely used channels like emails, Microsoft Teams etc.

Documentation templates

Documentation needs range from quick summaries and reports that provide high-level overview to comprehensive documentation that covers in-depth details. Create documents that represent not only the Azure resources but its expenses, the exact location of resources, the resource groups, the resource types, and a lot more.

For eg: Executive team may need an expenditure summary, auditing team may be interested in security reports, IT management team may need deeper insights.

Azure Documenter provides default templates to start with easy and insightful documentation and to configure custom templates that will meet your needs.

What are the benefits of integrating Azure Documenter with Document360?

You could of course read any data from your Azure environment and document it to Word or PDF. Once you create the documents you might want to share it with the wider community. You may want to review the document to keep track of the environments and send it to a colleague to review. Though you have SharePoint, office365, or other online platforms to share your files, they demand manual efforts and resources to store, maintain versions, distribute, and access control.

Maintaining versions on word or PDF and sharing them with stakeholders through shared folder access becomes too difficult to manage. As teams grow, managing access control of documents to a wider audience is tedious.

Document360, an online Knowledge Base authoring, and publishing platform is used to publish the documentation generated by Azure Documenter. It provides online publishing, restricted access to the documentation, and archiving capabilities.

Managing Azure Event Grid Dead-Letter events using Cerebrata

Modhana — Sun, 13 Feb 2022 15:56:22 +0000

Introduction

Azure Event Grid allows you to easily build applications with event-based architectures. In this blog, we will understand how Cerebrata Cerulean will help us in managing the dead-letter events of the Event Grid Subscriptions that are stored in the storage blobs.

Event Grid Topics and Subscriptions

Event Grid architecture is mainly composed of two components. They are Event Grid Topics and Event Grid Subscriptions. The event sources send the events to the Event Grid Topics. The Event Grid Topics then transmit the events to the Event Grid Subscriptions that are created within the Event Grid Topics based on the filtering configured in them. The Event Grid subscriptions then forward the events to the configured endpoints.

Dead-Letter Events

As we saw earlier, the events are sent to the configured endpoint or destination by the Event Grid Subscriptions. There are some cases in which the Event Grid Subscription cannot deliver those events to the configured endpoint. In such cases, the events will be dropped after the configured number of retries. In such cases, to prevent the loss of events, we can enable dead-lettering in the Event Grid Subscription.

Let us consider a scenario, where a Service Bus Queue is configured as an endpoint to the Event Grid Subscription. When the configured Service Bus Queue gets disabled due to some reason, then the Event Grid Subscription cannot deliver the events received from the Event Grid Topic. In this case, the Event Grid subscription will retry to send these events based on the Retry Policy configured. Once the maximum number of retries is exhausted, the Event Grid Subscription will lose the events. To overcome this event loss, we can enable dead-lettering on the Event Grid Subscription. We can configure any of the storage containers in the Azure Storage account as the dead-letter destination. The events that cannot be delivered to the endpoint will be stored as storage blobs.

Managing the Dead-Letter Events

It is not possible to directly view the dead-letter events of an Event Grid Subscription in the Azure Portal. To access the dead-letter events, we will have to navigate into the configured Storage Account container and traverse through into the folder where the events are stored which is quite a long process. To resubmit these events back to the Event Grid Topic, we will have to use the .Net SDKs or REST APIs to read the events from the storage blobs and to resubmit them back to the Event Grid Topic.

Managing the Dead-Letter Events through Cerebrata

Cerebrata overcomes the pain point of managing the Event Grid Subscription dead-letter events. It is possible to view and manage the dead-letter events by modifying and resubmitting them back to the respective Event Grid Topic and deleting the events based on the business requirements. We will see how Cerebrata simplifies the dead-letter event management here.

Steps in Viewing the Dead-Letter events

• Navigate into the required Event Grid Topic in the corresponding Subscription. The topic can be either a System or a Custom topic.
• Select the required Event Grid Topic to view the subscriptions created within the Event Grid Topic.
• Double click on the required Event Grid Subscription to view the dead-letter events in that Event Grid Subscription.

• All the dead-letter events stored as storage blobs will be listed in the opened tab.
• Select the required event to view the Event details.
• The Event details will be displayed as shown below. The Event details include Event Id, Subject, Event Type, Event Time, Data Version and Event Content.

• We can also find the dead-letter reason and the last delivery outcome of the event.

Steps in modifying and resubmitting the dead-letter events

The dead-letter events of the Event Grid Subscriptions belonging to the Event Grid Custom Topic can be resubmitted once the endpoint configured is ready to receive the events. This can be done at ease using Cerebrata. Follow the below steps to Modify and Resubmit the required events

• Select the required dead-lettered event.
• Select the Modify and Resubmit button.

• The Event details will be displayed in a pop up as shown below.

• The event content can be modified based on the requirements.
• We can also modify the advanced properties like Subject, Type and Data version of the event.
• Select the Save button to complete the resubmission process.

Steps to be followed to delete the Dead-Letter events

• Navigate into the required Event Grid Subscription where the dead-letter events will be displayed.
• Select the required dead-letter events to be deleted.
• Select the Delete button.

• Select the Delete button in the confirmation popup to complete the deletion process.

Other Event Grid Features

In addition to the dead-letter event management of the Event Grid Subscriptions, Cerebrata Cerulean also supports the following features.

• Managing the Tags of Event Grid Custom Topics and Event Grid System Topics
• Managing the Keys of Event Grid Custom Topics

Conclusion

In this blog, we learned about how Cerebrata Cerulean simplifies the process of dead-letter management of Event Grid Subscriptions. Apart from Azure Event Grids, Cerebrata Cerulean enables you to manage your Azure Cosmos DB accounts (SQL API, Gremlin API, and Table API), Storage Accounts, Service Bus Namespaces, Cognitive Search Service accounts, Redis Cache accounts, and much more. It is also cross-platform so that you can manage your Azure resources from a platform of your choice – Windows, Mac, or Linux. Please visit https://www.cerebrata.com to learn more.

Modify and Resubmit Messages in Azure Service Bus Queues and Topics/Subscriptions

Modhana — Sun, 06 Feb 2022 05:49:59 +0000

In this post we will see how Cerebrata can be used efficiently to manage the modification and resubmission of messages in the Azure Service Bus Queues and Topics/Subscriptions.

Problem

There may be a scenario where the messages that sent to Azure Service Bus Queue or Topic/Subscription may not delivered to the destination due to some technical issues or the messages was not processed successfully. What you would want to do is to resend the same message again with or without modifications. To send the message again you must create a new message from the beginning even if the message has some minor changes. It would consume more time and effort to recreate the same message.

Solution

Cerebrata offers the modify and resubmit functionality to overcome the above issues. Using this functionality, you can modify the contents and/or properties of the same message and the modified message can be resubmitted to the respective Queue or Topics with a single click.

Note that the messages which are in the dead lettered Queue or Subscriptions can also be resubmitted to the active Queue or Topic respectively. To modify the message in a Service Bus Queues or Subscriptions you must open a message from the active or dead lettered Queue or Subscription > open the message which is to be modified > select the Modify & Resubmit button in the bottom panel.

Using Cerebrata, you can modify both message contents and its properties before resubmitting. Cerebrata also provides an option to add or delete new custom properties to the message. Cerebrata provides an additional option for scheduling the enqueue time for the message that is to be resubmitted, where you can schedule the time so that the message will be enqueued into the Queue or Topic only at the scheduled time.

To schedule the enqueue time and to add or delete the custom properties you must open a message from the active or dead lettered Queue or Topic > open the message which needs to be modified > select the Modify & Resubmit button in the bottom panel > click enqueue time button to schedule the time and click custom properties button to add or delete the custom properties.

Summary

In this post we saw how you can modify and resubmit messages to an Azure Service Bus Queue/Topic using Cerebrata. Cerebrata provides most comprehensive set of features for managing Service Bus Namespaces, Queues, Topis, Subscriptions and messages. You can learn more about it on our website at https://cerebrata.com/features/azure-service-bus.

Other than that, Cerebrata has best-of-the-breed management features for Azure Storage, Cosmos DB, Cognitive Search, Redis Cache and more. You can learn more about the available features on our website at https://www.cerebrata.com/.

Azure Cosmos DB SQL API – Document Analytics

Modhana — Wed, 19 Jan 2022 03:27:16 +0000

Introduction

In this blog, we will understand the basics of Cosmos DB partitions and the use of Cerebrata in visualizing the data stored in Cosmos DB SQL API containers based on the partition keys.

Logical Partitions

A logical partition consists of a set of items that have the same partition key. For example, in a container that contains data about customer addresses, all items contain a state property. You can use state as the partition key for the container. Groups of items that have specific values for state form distinct logical partitions. When new items are added to a container, new logical partitions are automatically created by the system.

There is no limit to the number of logical partitions in your container. Each logical partition can store up to 20 GB of data. Good partition key choices have a wide range of possible values.

Physical Partitions

A container is scaled by distributing data and throughput across physical partitions. One or more logical partitions are mapped to a single physical partition internally. Typically, smaller containers have many logical partitions, but they only require a single physical partition. Unlike logical partitions, physical partitions are an internal implementation of the system, and they are entirely managed by Azure Cosmos DB.

The number of physical partitions in container depends on the following:

The number of throughput provisioned (each individual physical partition can provide a throughput of up to 10,000 request units per second). The 10,000 RU/s limit for physical partitions implies that logical partitions also have a 10,000 RU/s limit, as each logical partition is only mapped to one physical partition.
The total data storage (each individual physical partition can store up to 50GB of data).

Choosing a Partition key

There is no limit for the number of logical partitions in a container. Each logical partition can store up to 20GB of data. It is not possible to insert more data into the same logical partition once the 20 GB limit is reached. Let us consider a scenario, where we need to store the address of all the customers in a Cosmos DB SQL API container. The sample document to be stored will be of the following format.

{

“id”:”CS001”,

“firstName”: “John”,

“lastName: ”Smith”,

“addressLine1”: ”132, My Street”,

“addressLine2”: ” Kingston”,

“state”: ”New York”,

“zipCode”: “12401”,

“country”:”United States”

}

The partition Key path chosen for this container is “/state”. This will hold good when the population of the state is small or normal. But when the population of the state is large, then we will have a greater number of records with the same partition key and the size of the logical partition will grow enormously and may reach the 20 GB limit.

So, it is very important to note that the partition key must be chosen in such a way that it holds a wide range of values. So, in the above case, it will be good if the partition Key chosen is zipcode.

Document analytics in Cerebrata

In Cerebrata, it is possible to visualize the data stored in each logical partition within a container. This is available through the document analytics feature. Through this feature, we will be able to view the total size of the documents in each logical partition. This will help us in understanding the trend of the records that are stored in each logical partition.

Steps in viewing the Cosmos DB SQL API document analytics
Navigate into the required Cosmos DB SQL API container within the desired database

Select Document analytics from the context menu of the container.

The document analytics will be displayed in both graphical format and tabular format.

Capabilities in Document analytics

Search Criteria: It is possible to retrieve the documents that are modified within the provided data time interval. This can be done be selecting the Search Criteria option.

Document count: It is possible to visualize the total number of documents in a logical partition by grouping the records by document count.

Document size: It is possible to visualize the total size of documents in a logical partition by grouping the records by document size.

Features in Document Analytics

Export Chart: It is also possible to export the document analytics as an image file. This can be done through the Export chart option.

Export data: It is also possible to export the document analytics as CSV or Excel file. This can be done by using the Export data option.

Other features for Cosmos DB SQL API

Cerebrata also provides some other interesting features for Cosmos DB like management of databases, containers, stored procedures, functions, and triggers of containers in a Cosmos DB account. It is also possible to query the documents in a container, insert a new document, update, and delete the existing documents. It is also possible to copy the documents from one container to another container.

Conclusion

In this blog, we understood how Cerebrata can be used to visualize the logical partitions in a Cosmos DB SQL API container and the basics of logical and physical partitions.

Apart from Azure Cosmos DB SQL API, Cerebrata enables you to manage your Azure Cosmos DB accounts (Gremlin API and Table API), Service Bus Namespaces, Cognitive Search Service accounts, Redis Cache accounts, and much more. It is also cross-platform so that you can manage your Azure resources from a platform of your choice – Windows, Mac, or Linux. Please visit https://www.cerebrata.comto learn more.