DEV Community: Harinder Seera 🇭🇲

From Bedroom Disasters to Cloud Resilience: Explaining AWS DR Strategies To Anyone

Harinder Seera 🇭🇲 — Sun, 10 Dec 2023 22:47:11 +0000

Disclaimer: While this article simplifies DR strategies, it's important to remember that DR is a complex topic with many nuances.

In recent years, I've come to realise that concepts many of us in information technology take for granted aren't always common knowledge among everyone working in the technology industry. It highlights an ongoing need in the tech industry to explain complex ideas clearly to broad audiences with little to no exposure to the ever-evolving technology world.

I have recently started writing articles to explain complex ideas in a way everyone can easily understand. My goal is to make technical subjects understandable to anyone. In this article, I aim to explain Amazon Web Services (AWS) disaster recovery strategies through a household analogy. By relating cloud computing concepts to familiar scenarios, I hope to present AWS concepts in a simple yet insightful way for all readers. I hope to bridge knowledge gaps and demystify cloud technology for the non-techie person.

To begin with, below are different DR strategies that can be implement in an AWS cloud. Namely:

Backup and Restore
Pilot Light
Warm Standby
Multi-site Active/Active
Multi AZ deployment

Ref

Now we will go through the house analogy to explain them individually. We will start with the Multi AZ deployment strategy.

Single AWS Region - Multi AZ Deployment Strategy

If you are wondering what a region is, I have written an article called "AWS Is A Zoo: Anyone Can Navigate the Cloud Jungle!" It explains what a region is using the Zoo analogy.

Back to our DR strategy, Imagine you have a house with three bedrooms. You and your partner occupy one room, while your two kids have their separate rooms. Unfortunately, while you were all out for dinner, the roof of one of your child's bedrooms collapses (as shown in the image above). Luckily, as your house has three rooms, with some adjustments like moving a bed or using an air mattress, your children can share a room temporarily while the repairs are being made to the other room. You won't have to make many changes to address the issue of where your kid would sleep.

This scenario resembles having a single AWS region with multiple Availability Zones (AZ). In case of a disaster such as a natural calamity or a technical failure that destroys one physical data centre (AZ), having your workload distributed across various Availability Zones within the same AWS Region can assist you in weathering the storm, whether it's a natural disaster or a technical glitch. If one of the AZs is unavailable, at least your workload can continue operating in the other AZs.

Multi AWS Region Strategy

Let's imagine you have two homes: a primary residence and a vacation home. These will be used to explain the remaining AWS disaster recovery strategies.

Backup and Restore Strategy

In your primary house, you have a fireproof vault where you regularly keep copies of your important documents such as birth certificate, driving licence and passport. In the event of a fire, you can retrieve these documents to identify yourself or use them for other purposes, such as home insurance claims.

Similarly, on AWS, the back and restore (DR) strategy involves regularly backing up your data to the cloud. This involves regularly backing up your data to the cloud. In the event of any disasters, you can use these backups to restore lost data and applications. This strategy can be applied to both single-region and multi-region implementations. For a single region, you can copy the data to a different account, while for a multi-region, you can copy the data into your secondary region and retrieve it when needed.

Pilot Light Strategy

Imagine a vacation home that you plan to use only occasionally. It's not fully furnished or equipped for daily living; it has a minimal/basic structure in place, just like in the picture above. You could make it habitable with some effort, but it's not ready for immediate occupancy.

This analogy closely resembles the pilot light disaster recovery (DR) strategy on AWS. The infrastructure in the recovery region is minimalistic, similar to the basic structure of a vacation home. It provides the core components needed to run your applications but lacks the fully configured environment required for production workloads. When a disaster occurs, you must provision additional resources, configure applications, and perform other manual tasks before the recovery region can fully handle production traffic.

Warm Standby Strategy

With the warm standby approach, your vacation home is fully built but needs to be fully furnished and stocked. It is ready for immediate use, though it may not have all the amenities and personal touches of your primary residence.

Similarly, the warm standby DR strategy on AWS involves maintaining a scaled down, but fully functional, copy of your production environment in the recovery region. This replica is kept up-to-date with data changes, but it may not be running at full capacity or serving all application components. When a disaster occurs, you can quickly failover to recovery region, minimising downtime.

Multi-Site Active/Active Strategy

With the active/active approach, both your primary residence and vacation home are fully equipped and ready for daily use. You can comfortably switch between them depending on your needs or preferences. You can have your family living in your vacation home while you reside in your primary home or vice versa.

The multi-site active/active DR strategy on AWS mirrors this concept. You have two production environments running simultaneously in different regions, each serving a portion of your application traffic. This strategy offers the highest availability and lowest recovery time objective but comes with the highest cost and complexity.

Closing Thoughts

In this article, I aimed to explain key AWS disaster recovery concepts in an easy-to-understand manner. By comparing cloud computing strategies to everyday household scenarios, my goal was to make complex technical ideas simple and accessible. Though concepts such as Availability Zones and recovery time objectives may be familiar to those with a technology background, they can be challenging for those without a tech background.

I hope that these simplified explanations shed light on AWS disaster recovery in a way that resonates with everyone. If this piece has helped demystify even one core concept for you, then it has achieved its purpose.

Thanks for reading!

If you enjoyed this article, feel free to share it on social media 🙂

Say Hello on: Linkedin | Twitter

Github repo: hseera

Observability - Licencing Challenges and Tips To Consider

Harinder Seera 🇭🇲 — Sun, 10 Dec 2023 22:35:54 +0000

Disclaimer: The information I am sharing is based on my personal experience and may differ from the experiences of others.

In many of my past roles, whether as a new team member or as a consultant, I wasn't actively involved in the negotiating process of acquiring or renewing monitoring or observability tools. However, in recent years, I've been actively involved in the process, such as Request for Information (RFI), Request for Proposal (RFP), and Request for Quote (RFQ).

For those unfamiliar with these processes, they can be complex and challenging due to various factors. Understanding these factors is crucial for navigating the license renewal process or acquiring a new observability tool for your organisation.

Varied Pricing Models - Observability vendors offer different pricing models that can include per-host, per-container, per-metric, per-active service, per-GB of data ingested, and per-user, among others. This wide range of pricing models can make it challenging to compare prices among vendors and determine the right tool that fits your requirements and budget. Compared to other types of tools I have used, observability tools have the most diverse pricing models. I have previously written a post that outlines which vendor uses which pricing model.
Unpredictable Data Volume - The volume of observability data can significantly vary based on application usage, traffic patterns, and external factors like an organisation's maturity stage. Startups and growing organisations with acquisitions and mergers may experience unpredictable data volume growth. This variability makes it challenging to estimate costs when negotiating pricing.
Hidden Costs and Overage Fees - Some vendors may have hidden costs, such as data retention fees, add-on charges for advanced features, or overage fees for exceeding usage limits. These hidden costs can inflate the overall cost of observability, complicating accurate cost forecasting. This was another point I covered in my previous post.
Lack of Transparency - Pricing structures can be complex and opaque, making it difficult to understand what you are paying for and how usage translates into costs. This lack of transparency can lead to confusion and frustration, especially when unexpected charges arise. In recent times, I have observed that vendors are taking steps to make the pricing structures more transparent.
Contractual Complexity - This challenge is not unique to observability tools but applies to other types of services as well. Contracts can be comprehensive and intricate, including different pricing terms, usage limitations, and service-level agreements. It is essential to fully grasp these contractual terms to avoid any unexpected situations and ensure that your organisation's expectations are met. For example, some organisations may have data residency limitations, which can create licensing difficulties. If your organisation has legal and risk teams, it is advisable to have them review the contract.
Education - Different teams (for example sourcing) involved in the process may be familiar with pricing models such as per-user or per-host licensing, as these are some of the most common pricing models used in the industry. However, observability tools often employ alternative pricing models or, in some cases, a combination of different pricing models. This necessitates team training on the diverse pricing structures of observability tools. Moreover, the complexity of observability tool pricing further complicates the education process for the teams. For example Per-GB pricing for ingested data may be based on the number of spans, events, and traces ingested. It is important to educate teams on what these terms mean so they are better equipped for future and also can help you negotiate a positive outcome.

Below are some of the tips you can consider when trying to address some of these challenges, whether you are renewing a tool or acquiring.

Requirements - It is important to clearly define your observability needs and the features required to address them based on the type of process you are undertaking. For instance, the level of detail you will need to communicate with the vendor will differ in an RFI process compared to an RFQ or RFP process.
Request Detailed Pricing Breakdowns - Seek detailed pricing breakdowns from vendors, clearly outlining all costs, including base fees, add-on features, data retention fees, overage fees, and any other applicable charges. And ask as many questions as you can to make sure the pricing breakdown makes sense.
Negotiate Flexible Terms - Explore negotiating flexible pricing terms that better align with your organisation's usage patterns and growth projections. This could involve tiered pricing plans, volume discounts, or adjustments based on usage. For example you may have a tiered pricing based on the amount of data you ingest. For the initial 100GB, the rate is set at $100 per GB, while for the subsequent 100GB to 200GB range, the pricing adjusts to $90 per GB.
Evaluate Open-Source Alternatives - Consider looking into open-source observability tools as an alternative. Although they may require more effort to set up and maintain, they can help you save on licensing costs. However, keep in mind that operational costs may increase. It is essential to exercise caution when exploring this option. This could be a viable choice for startups not in a position early on to invest in licensing costs for an observability tool.
Regularly Review Usage and Costs - Monitor your observability tools' usage and associated costs closely. Identify areas to optimise usage and reduce costs by adjusting data retention settings, avoiding unnecessary data transmission, modifying default signal settings, or renegotiating contract terms if usage patterns change significantly. Partner with a vendor who understands your business and is responsive to changes impacting costs.
Pricing Model - Wherever applicable, I advise to maintain a consistent pricing model across your organisation. For instance, if you are using the same tool, it is better to have a uniform pricing model for all departments instead of having different pricing models for different departments. This approach can help avoid complexities during contract renegotiation, as you won't have to deal with multiple pricing models.
Cloud Provider Observability Solution - Consider leveraging the observability solution offered by your cloud provider (i.e. AWS, Azure, Google). Many organisations benefit from volume discounts through enterprise agreements, which can significantly reduce costs. Additionally, most cloud providers utilise pay-as-you-go pricing, ensuring you only pay for the resources you utilise.

Hopefully this post has given you things that you need to consider and steps you can take when acquiring or renewing an observability tool.

Thanks for reading!

If you enjoyed this article, feel free to share it on social media 🙂

Say Hello on: Linkedin | Twitter

Github repo: hseera

AWS Is A Zoo: Anyone Can Navigate the Cloud Jungle!

Harinder Seera 🇭🇲 — Mon, 20 Nov 2023 09:55:58 +0000

In my line of work, I often run into people who aren't familiar with AWS. It's understandable; cloud computing is an immense and constantly evolving realm. This adds a challenging yet fascinating aspect to my job, pushing me to think creatively and find effective ways to explain AWS concepts that resonate with everyone.

In this article, I'm going to use a simple analogy to illustrate some fundamental AWS concepts. By the end, I hope that even those with no prior AWS experience will have a basic grasp of the key terms. So next time someone is talking about different regions and availability zones and how AWS handles resilience, you won't need to ponder what an availability zone is.

For this illustration, we'll use Australia as an example. However, you can replace Australia with any other country or continent, too, for explanation purposes.

AWS: The Animal Kingdom Analogy

Picture Australia as a vast and diverse continent, a thriving animal kingdom teeming with unique creatures and experiences that offer a wealth of opportunities for kids and families to explore and enjoy. In a similar vein, Amazon Web Services (AWS) stands as the world's most comprehensive and widely adopted cloud platform, delivering over 200 fully featured services from data centers located across the globe.

Regions: The States and Territories of the Animal Kingdom

Australia, as a continent and, in this case, as an animal kingdom, is divided into states and territories, each with its own laws, governance, and unique characteristics. Similarly, AWS is divided into regions, which are distinct geographical areas spread across the globe.

Availability Zones: The Zoos of Each State or Territory

Within each region, there are Availability Zones, which are like individual zoos. These Availability Zones are geographically separate from each other, providing redundancy and fault tolerance. If one Availability Zone goes down, the others will still be operational, ensuring that your cloud applications remain available. This analogy is similar to Victoria state having four zoos: Healesville Sanctuary, Kyabram Fauna Park, Melbourne Zoo, and Werribee Open Range Zoo. If one zoo is temporarily unavailable on any given day, parents can still take their children to the other zoos. Typically, an AWS Region will have at least three Availability Zones.

AWS Services: The Animal Enclosures

Just as the Melbourne Zoo houses a variety of animals (Elephants, Lions, Tigers, and Wombats, to name a few), each with its own enclosure (just like the above image), AWS offers a wide range of services that organizations can use to build their cloud-based solutions. These services, such as EC2, S3, CloudWatch, and Lambda Functions, are like the building blocks. Think of different animal enclosures in a Zoo as different AWS services available in a region.

Individual Resources: The Specific Animals

AWS resource is an entity that you can work with. They are like the animals within a zoo enclosure. For example, an application might require two EC2 resources and one S3 resource, just like seeing two elephants and one lion in the Melbourne Zoo, just like the picture above. Each resource serves a distinct purpose, contributing to the overall functionality of the cloud-based system.

Common AWS Services Explained Using the Zoo and Animal Analogy

Here are some of the most common AWS services explained using the zoo and animal analogy:

Elastic Compute Service (EC2) and Instance types: EC2 is like lions in an enclosure. Instance types are like lions, lionesses, and cubs, each with its own characteristics. You have compute, memory, and general-purpose instance types to name a few. AWS EC2 is an on-demand computing service on the AWS cloud platform.
Simple Storage Service (S3): Amazon S3 stores and retrieves any amount of data at any time, from anywhere on the web. Think of this as food storage for animals in a zoo.
Identity and Access Management (IAM): IAM is like the security system of the zoo. It controls who has access to different areas, ensuring only authorized users can enter specific enclosures. AWS IAM lets you define individual users with permissions across AWS resources and enables Multi-Factor Authentication for privileged accounts, including options for software- and hardware-based authenticators.
AWS CloudWatch: Think of it as the zookeeper who monitors the animals' health, behavior, and interactions. They track key metrics like food consumption patterns, growth rate, and activity level. Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real-time.
AWS Lambda Functions (Serverless): Think of this service as a special event in a zoo. Consider a portable ice cream stall that appears on a scorching summer day, delighting visitors with refreshing treats. As the day concludes and temperatures cool, the stall vanishes, leaving no trace of its presence. Similarly, AWS Lambda functions pop up when needed, executes the required code efficiently and then terminates until the next invocation.

By drawing parallels between the familiar world of a zoo and the intricacies of AWS, I hope to make cloud computing more approachable for newcomers. By understanding the fundamental concepts of regions, Availability Zones, and services, you can confidently engage in discussions about AWS without feeling overwhelmed. As your cloud computing knowledge expands, you'll be able to delve deeper into AWS's vast and ever-evolving ecosystem and come up with different metaphors to explain the concepts better than I did.

After reading the article, Can you think of other AWS services that can be explained using zoo analogy?

Thanks for reading!

If you enjoyed this article, feel free to share it on social media 🙂

Say Hello on: Linkedin | Twitter

Github repo: hseera

Overcoming Butterflies

Harinder Seera 🇭🇲 — Thu, 09 Nov 2023 09:40:48 +0000

 "...His palms are sweaty, knees weak, arms are heavy
         There's vomit on his sweater already, mom's spaghetti
        He's nervous, but on the surface, he looks calm and ready..." 
                       - Eminem (from Lose Yourself song)

I have a little experience in public speaking, primarily in person. I only do one or two public speaking engagements yearly, but I do virtual speaking more often. Whenever I have to speak in public, I get nervous, and my palms become sweaty, my knees weak, and my arms heavy - just like in the lyrics of Eminem's song "Lose Yourself." I start having butterflies in my stomach.

In this post, I want to share my experience presenting at the AWS Community Summit in Thailand last year. I presented to over 120 people from different countries and was surprised to find myself feeling relaxed. Looking back and thinking about it now, a few things helped me. Some of them were intentional, and others were not.

I hope this post will inspire you to take the risk of public speaking, and hopefully, you can leverage some of the things I outlined below to help you deliver your speech in person.

Practice, practice, practice! - Each talk was allotted 20 minutes. As a speaker, I needed to rehearse my speech and enhance my delivery to stay within the given time. I made it a daily routine to practice my speech early in the morning, multiple times in front of the mirror while timing myself. With each run-through, I aimed to improve my delivery. I found that structuring my slides in a clear and concise way helped me to stay on track during the actual talk. Practicing multiple times meant that my words flowed naturally without hesitation when it came time to speak.
Meet And Greet - On the first day of our arrival, we had a welcome dinner for all the attendees. During dinner, I took the opportunity to converse with as many people as possible. This was not intentional. Looking back, the act of meeting as many people as possible transformed my scary 'public' speaking into a more natural, relaxed 'personal' speaking. It was as if I was talking to my friends. I also had the opportunity to explore the city with other attendees, which helped me become more comfortable around them.
Rehearsal Opportunity - As a speaker, I was allowed to rehearse my speech early in the morning in an empty room. This allowed me to check my slide deck and ensure it would appear correctly on the screen. This opportunity also helped me get a feel for the room and visualize myself on stage, all while avoiding the anxiety that comes with having a crowd of people watching. I didn't want to find myself in a situation where I was unable to use the clicker properly in front of an audience.
Eating light - I was scheduled to give a talk on the last day of the summit, after lunch. To ensure that I gave the best talk possible, I decided to follow the advice of several articles and avoid eating a large meal beforehand. I found that eating light helped me stay focused and prevented my vocal cords from being affected by any beverages I consumed. At the same time, I made sure not to skip a meal entirely, as I didn't want my stomach to start rumbling during my talk. Overall, it was a wise decision to eat light.
Arriving Early - I arrived early in the room where I was scheduled to speak after lunch. The room was quiet and empty as the audience was still having lunch. This allowed me to get on the stage again and feel the space. Moreover, I had a chance to speak with the AWS team, who were assisting all the speakers and ensuring that the event went smoothly. Talking with them helped me feel more confident and relaxed. They reassured me that I would be alright, and I just had to get on the stage and enjoy the moment.
Starting With A Story - Starting my talk with a story turned out to be incredibly helpful, especially during the first few minutes when I was trying to gain control of my nerves. Although I still felt a little anxious, it wasn't to the extent that I started shaking or forgetting what I needed to say. Additionally, it helped grab audience attention after lunch.
Having an Anchor - During my presentation, I ensured that I established a connection with the audience by making eye contact. However, I found myself looking back at the table where my friends from Australia and New Zealand were seated, as they were my anchor point to calm my nerves. Despite making eye contact with everyone in the room, I regularly looked at their table to help myself feel at ease.
"Blue Screen Of Death" And Just Breath - It's important to remember that even the most successful people experience setbacks. For example, Bill Gates famously experienced the Blue Screen of Death. So, instead of being hard on myself, I asked myself, "What could potentially go wrong?" Remember that even if I messed up, the world will keep spinning and I will still get another opportunity. There will always be a next time to do better, so take comfort in that.

I have had multiple chances to speak in front of the public ever since. With each opportunity, I observed a significant improvement in my skills and a growing ability to reflect on areas that require further growth.

Final Thoughts

If someone like me, who is usually shy at public speaking, can do it, you can do it, too. Don't be afraid to try it, as it may not be as scary as you think. I am going to end the post with...

"...You only get one shot, do not miss your chance to blow.
       This opportunity comes once in a lifetime..." 
             - Eminem (from Lose Yourself song)

Thanks for reading!

If you enjoyed this article feel free to share on social media 🙂

Say Hello on: Linkedin | Twitter

Github repo: hseera

Icicle Graphs - Easy Way To Visualize Cloud Cost Hierarchical Relationships

Harinder Seera 🇭🇲 — Mon, 31 Jul 2023 10:28:35 +0000

Thanks to Brendan Gregg's work in visualizing stack traces, flame graphs have become a popular tool for analysing performance. They provide a simple and intuitive way to understand complex data by showing the relationship between function calls and resource consumption. The graph makes it easier to identify areas of improvement and optimize various systems, from CPU and memory to I/O subsystems. Flame graphs have significantly impacted software performance engineering, enabling developers to fine-tune their applications. Similarly, there is a need for a similar approach to tackle the challenge of managing Cloud costs across multiple accounts, teams and resource usage.

Managing cloud costs can be challenging, especially when it comes to understanding the hierarchical relationships between accounts, resources, and team/cost centers. While there are cloud cost management tools that help visualize cloud costs and identify areas of overspending, they often fall short when it comes to displaying this information clearly and comprehensively. It becomes even more challenging when multiple teams work across different services and manage them across different accounts. But don't worry - there's a solution! Icicle graphs are a powerful tool that can be leveraged to break down cloud costs hierarchically, making it easier to understand and manage your spending.

What is an Icicle Graph?

Icicle graphs, a type of rectangular sector visualization, are variation of flame graphs. They are highly effective in presenting hierarchical data that flows from the root to the leaves in four directions: up, down, left, or right. With this feature, icicle graphs make it easier to identify clusters and comprehend their hierarchy level. Moreover, these graphs are great in recognizing relationships in data.

It is worth noting that icicle graphs may not be the most suitable option for visualizing large hierarchies as it can be challenging to examine individual data points in detail. However, when dealing with smaller hierarchies such as AWS cloud bill across multiple accounts, these graphs are incredibly effective and particularly useful in identifying correlations and clusters within the data.

AWS Bill Visualization Example Using Icicle Graph

Below is an example of hierarchical cost data of AWS services. It can become challenging if you want to identify cost patterns and relationships across different dimension. This is where visualization such as Icicle graphs become useful.

Check out visualization below of cloud costs using an icicle graph to show hierarchical relationships. The leftmost node represents the organization, and it has five child nodes representing the accounts (LinkedAccount1 to LinkedAccount5) sorted by the highest spend account at the top. Each account node then breaks down further to AWS product/service nodes, with the service that had the highest spent at the top. This pattern continues until you get to the usage type, allowing for a clear breakdown of cloud costs.

Here is the video visualizing the AWS cloud bill using icicle graph.

Thanks for reading!

If you enjoyed this article feel free to share on social media 🙂

Say Hello on: Linkedin | Twitter

Github repo: hseera

AWS CloudWatch Dashboard Builder - How To Use Guide

Harinder Seera 🇭🇲 — Wed, 04 Jan 2023 09:03:46 +0000

This is the last post in the CloudWatch Dashboard Builder series. In this post I walk you through on how to use the tool. I recommend you read the post on namespace template if you haven't already. It will help you better comprehend the tool. You will also be able to create your own namespace template for use with the tool.

Download

The executable for the AWS CloudWatch Dashboard Builder can be found here.

If you like the tool, please rate it on github. Also, if you spot a bug, please file an issue so that I can fix it. Make sure to download the latest version of executable as it may have new updates.

Workbench Configuration

When you run the CloudWatch Dashboard Builder executable, you will be presented with the Config screen.

In the tool, you have two options for connecting to AWS services using credentials. The first step is to manually enter credentials on the configuration tab. Another option is to keep your credentials in the following location on your Windows PC.

/Users/[username]/.aws

Example:

For the article, we'll enter our credentials into the text box and connect.

Once the connection is established, select the "Cloudwatch Dashboard Builder" tab. Here and the "Namespace Query Template" tab is where the magic happens. The following are the various components that comprise the screen.

1. Namespace

A namespace is a container for CloudWatch metrics. Metrics in different namespaces are isolated from each other. Namespace list is automatically populated when "Cloudwatch Dashboard Builder" tab is clicked. List can also be refreshed by right clicking on the list and selecting "Refresh Namespace".

2. Template

A template is a namespace-contained container for CloudWatch metrics. Each namespace template reports on distinct metrics. Click on the namespace in the namespace list to see a list of all the templates in that namespace. For example, when you select an SQS namespace, it displays two different metric templates for it.

3. Template detail

The template detail describes what the purpose of the template is. What is it meant to report? It also shows the template query that is associated with it. For example, selecting the "Top 10 queues by age" template gives you detail about the template and the SQL query associated with it.

4. Region Name

The region name displays a list of all AWS regions that are available around the world. To view the list, click the "List Regions" button.

5. Existing Dashboards

If the AWS account already has existing CloudWatch dashboards, you can view them my right clicking on the "Existing Dashboard" list and selecting "Show Existing Dashboards"

6. Dashboard Queries

To add queries to the dashboard, highlight the query, right-click on it, and select the "Add Query" option. This will add the query to the query list.

You can remove the query from the list by selecting it, right-clicking, and clicking on the "Remove Query" option.

7. CloudWatch Dashboard JSON

To generate the CloudWatch dashboard JSON, right click on the query list and click on "Generate Dashboard JSON."

At this point, you have two options. You can either copy the JSON and use it via the CLI or AWS Console to generate the dashboard, or you can directly create it through the tool.

8. Create Dashboard

To create a dashboard through the tool, enter the dashboard name in the "Dashboard Name" text field and click on the "Create Dashboard" button. You will see a message in the console indicating whether the dashboard was successfully created or not.

9. Console

The console screen contains all errors, responses, and logs. The tool's "Save Output" option allows you to save console data for further analysis. To clear the console message, use "Clear Output". In the current version, the output is saved in the same folder as the executable. "output.txt" is the name of the output file.

10. Namespace Query Template

The Namespace Query Template tab allows you to load and edit the namespace query template. You can add new templates to existing namespaces, create new namespaces and templates, and delete or modify existing namespaces and templates.

Tool Caveats

There are a couple of caveats that need to be taken into account.

Currently there is only Windows executable shared on github. Since the code is open source, you could build one to run on Linux.
The present version produces only a time series dashboard.
Double quotes in the namespace template queries need to be escaped.

Finally, here is a video showing how to use the tool to generate a dashboard or how to use the JSON to construct a dashboard using the CLI or AWS Console.

AWS CloudWatch Dashboard Builder - Namespace Template

Harinder Seera 🇭🇲 — Wed, 04 Jan 2023 09:03:30 +0000

This is the second post in the CloudWatch Dashboard Builder series. In this post, I'll walk you through the Namespace query template, which is the foundation for the AWS CloudWatch Dashboard Builder tool

The namespace query template defines the metric queries for different AWS services. SQL expressions can be used in metric queries. Examples of queries with and without a SQL expression are provided below.

As long as a naming standard is followed, the namespace template can be customized to match your needs. Certain keywords must be retained when modifying or creating a new namespace template for usage with the Cloudwatch dashboard builder tool.

The root object of the namespace template is called "cloudwatch_template" and it is a keyword. Under the root object, you can have multiple child objects, which are called namespaces. Each namespace object has a "schema" name and different "templates". Templates are the unique queries for the namespace. The template contains three keys:

name: Name of the template. This name is used as the widget title in the Cloudwatch dashboard.
desc: A short description of the template. Tells what the query template does, reports or measures.
query: A query that will be executed in Cloudwatch to report on the metrics of the namespace.

The namespace template is located in the template folder. For queries that have double quotes in them, they will need to be escaped for the query to work correctly. So special care needs to be taken when updating or creating the template.

Here is another example of a complex namespace template.

Now that you're familiar with the Namespace template, we can go on to the last post, which explains how to use the tool.

AWS CloudWatch Dashboard Builder - Tool For SRE, Performance Engineers and DevOps

Harinder Seera 🇭🇲 — Wed, 04 Jan 2023 09:03:16 +0000

This is a three-part series. The first post gives background on why I developed the tool, and later posts explain how to use it.

For over five years, I have been using the AWS CloudWatch service for monitoring and troubleshooting application performance. Over that period, I have developed different Cloudwatch metric queries to help me quickly create CloudWatch dashboards or widgets for AWS services.

In the beginning, I used to write these queries in a notepad. From time to time, I would refer them to create a CloudWatch dashboard or widget if I needed them for a new project, client, or application. Also, occasionally, I lacked access to CloudWatch, so I had to ask a colleague to create a widget or dashboard. Before requesting them, I had to transform the query into JSON format. These reasons led me to create a tool for building CloudWatch dashboards.

The CloudWatch Dashboard Builder tool generates a JSON dashboard template with different metric queries. Queries can include SQL expressions as metrics. You can directly create the dashboard or copy the JSON and create it via the AWS Console/CLI.

The flexibility of this tool stems from the fact that it is powered by a namespace query template that you can change to meet your requirements. Next post in the series will explore Namespace template into more detail.

Here is a video showing how to use the tool to generate a dashboard or how to use the JSON to construct a dashboard using the CLI or AWS Console.

The benefits of this tool are:

Capability to choose different metric templates for different AWS Namespaces or a custom Namespaces.
Saves time in building the right CloudWatch dashboard for your needs.
You can directly create the dashboard from the tool or generate the dashboard JSON.
You can enhance the Namespace Query template by including your metric queries.

In next post we will explore namespace template. It will explain how to use it correctly with your own CloudWatch metric expressions.

AWS CloudWatch Metrics Insights - Query Builder For Performance Engineers, SRE and Operations

Harinder Seera 🇭🇲 — Thu, 22 Dec 2022 11:07:41 +0000

During my time as a performance engineer who was also involved in operations, I wished AWS had the ability to query Cloudwatch metrics for various AWS services using a SQL-like query language. However, it is never too late to have such an option.

AWS recently introduced a SQL-like capability for querying metrics data in CloudWatch. It is part of the Cloudwatch Metrics Insight feature. No, I'm not referring to the ability to query logs. This is used to query Cloudwatch metrics data. This feature is currently in preview mode.

I haven't used it much except for cost optimization work, but given my previous struggles with visualising metrics data, the SQL capability is a useful feature to have. For example, previously, if I wanted to show only the top 10 EBS volumes based on the amount of data written to them, I would have to query them using the AWS SDK. Other options would have been to use the AWS console, add all the volumes to a dashboard, and then filter them one by one. A time-consuming process to follow. With SQL, however, I can simply execute the following code to obtain the top ten volumes with the most data written to them:

SELECT SUM(VolumeWriteBytes) FROM SCHEMA("AWS/EBS", VolumeId) GROUP BY VolumeId ORDER BY SUM() DESC limit 10

Here is the visualization of executing the above query in Cloudwatch using the query editor.

To access the SQL editor/query builder, navigate to CloudWatch page and select "All Metrics" under Metrics header.

Here's another example, If you simply want to see the top 5 EC2 instances ranked by highest CPU consumption, run the following Cloudwatch query.

SELECT AVG(CPUUtilization) FROM SCHEMA("AWS/EC2", InstanceId) GROUP BY InstanceId ORDER BY AVG() DESC limit 5

I started using this feature when it was introduced late last year to help me with cost optimization activities I was doing.

Since this functionality is in preview mode, there are few limitation that you need to be aware off.

You can query only the most recent three hours of data.
A single query can process no more than 10,000 metrics.
A single query can return no more than 500 time series.
Each GetMetricData operation can have only one query, but you can have multiple widgets in a dashboard that each include a query.

I wish AWS had had this option 5 years ago. Life would have been so much easier. If you want to learn more about this feature, please refer to the following links:

Observability - Types Of Vendor Pricing Models

Harinder Seera 🇭🇲 — Sat, 17 Dec 2022 02:19:04 +0000

In the last 5 to 10 years, new Observability vendors have entered the market, including Honeycomb, Instana, Lightstep and Datadog. Similarly, traditional APM vendors such as Dynatrace, AppDynamics, and New Relic, as well as SIEM (and log management) vendors such as Splunk and Sumo Logic, have joined them in the Observability space too. Finally, large cloud providers like AWS offer their own observability solution. Each of them is attempting to address the observability issues that modern architecture presents by using Logs, Metrics, Traces, and Events.

One thing that stands out when you look at them is the various pricing models they use for their observability solutions. And if you are not familiar with their pricing structure, it can become a bit confusing and challenging.

This post is not intended to imply that Vendor A is superior to Vendor B. It is meant to be a guide to help you understand the various pricing models you will come across when dealing with observability vendors. The vendor you choose will be determined by your specific use case, tool experience, and budget. So far, I've come across four primary pricing models.

Charge per Ingestion (event/span/data)
Charge per Active Service
Charge per Host
Charge per User

Charge per Ingestion

Example of vendors that use this type of pricing model are Honeycomb, SumoLogic, and AppDynamics. One of the new ingestion-based pricing models that people will encounter is charges per event or charges per span.

So what is an Event or Span?

A span represents a unit of work, and an event is a structured record of a discrete action that occurs at a specific time. An event is a single operation, such as an HTTP request that was processed by a system. The Lightstep website has a simple representation of span, as shown below. I recommend reading the articles in the appendix to learn more about span, trace, and events to make sure you are comfortable with the concepts.

Source

Charge per active Service

An example of a vendor that uses this type of pricing model is Lightstep. A service is typically a microservice, which relates to a specific functionality. According to this model, you will only be billed for the number of active services. So if you deploy 10 instances of the same service, you will only be billed for one service.

Charge per Host

Examples of vendors that use this type of pricing model are Dynatrace, Datadog, Instana and Splunk. Each vendor defines what they means by a host. However, what I have experienced is that when a vendor says "host," they generally mean a server, VM, node (in the case of Kubernetes), service that reports host name metrics or all of them. Make sure you ask them this question as to what they mean by host.

Charge per User

An example of a vendor that uses this type of pricing model is New Relic. In this type of model, you are charged based on the type and number of users.

A few vendors use a simple pricing model. The majority of them have some variation of the models listed above. For example, some employ both a user- and data-ingested pricing model. As a customer, it pays to make sure you understand and are comfortable with the pricing model.

It is also important to be aware of any potential hidden costs. Some may charge you extra for high cardinality or longer data retention. Find out what the extra costs are to avoid having to explain exorbitantly large bills to your senior management.

Finally, I would advise vendors to keep their pricing models as simple as possible. The market for observability solutions is already overly competitive. Having a complicated pricing model won't help your cause.

Appendix:

Useful Cloud Resource Tags For Cost Optimization

Harinder Seera 🇭🇲 — Thu, 24 Nov 2022 10:59:42 +0000

If you are just getting started with cloud migration, there are a few basic things you can do to monitor and manage your cloud costs. If you use AWS, for example, you can set Budget alerts, use Trusted Advisor for cost optimization advice, AWS Cost Explorer, and so on. These are simple tasks that take little effort but can help you get insights into your cloud cost.

Organizations tend to purchase expensive third-party tools to help them manage their Cloud costs. These tools have there place and do provide benefits. However, cloud-native tools (the ones mentioned above) and a little Excel magic may provide you with more than enough information to help you monitor and control your cloud spending. To utilize their capabilities, you must have strong resource tagging standards in place. If you want to go Third-party tools path, they can also benefit from from the right resource tagging standards.

If you aren't sure where to begin with the tagging standard, the post details some of the tags you can use to get you started. The post is mainly focused on AWS but the standard outlined will also apply to other Cloud providers.

AWS tagging standards outline how tags should be used to identify and categories AWS resources. These standards help to ensure that tags are applied uniformly across AWS resources, making resource management and tracking easier. Tagging is a useful strategy to apply when it comes to tracking your cloud cost consumption.

Tags should be used only for the most important metadata. Rather than making all tags essential, define optional tags that explicitly state how your teams should implement them if they so desire.

Environment tag - Help determine cost differences among development, test/staging, and production.
Application/Service tag - Identifies the primary services the resource belongs to,allowing the organization to differentiate costs among the services that teams are running.
Cluster tag - Identify resource farms that share a common configuration and perform a specific function for an application. This can be an optional tag.
Application-Role tag - Help identify the primary role of the resource.
Name tag - Identify the resource using a friendlier identifier than the one given to you by your cloud services provider.
Business unit/Cost-Center tag - Defines where the costs of the resource should be allocated within your organization. This will allow for correct cost allocation within your billing data.
Resource Owner/Support Owner tag - Help identify the individual/team responsible for the resource.

Some optional tags you may consider defining and not necessarily related to cost optimization include:

Compliance tag - An identifier for workloads that must adhere to specific compliance requirements Identify the type of data contained on the resource and whether it needs to adhere to specific compliance requirement such as PCI DSS compliant.
Confidentiality tag - Identify the amount of scrutiny that needs to be applied to the security level of the resource.
Automation tag - Used to select particular versions of resources to archive, update, or remove, as well as to opt in or out of automated actions.

I hope this post has given you a basic knowledge of why tagging is important, as well as some simple tags that you may start using to aid you on your cost optimization journey.

Thanks for reading!

If you enjoyed this article feel free to share it on social media 🙂

Say Hello on: Linkedin | Twitter | Polywork

Blogging: Dev | Hashnode

Github: hseera

Visualizing Cryptocurrency Data With AWS Serverless Technology

Harinder Seera 🇭🇲 — Sat, 19 Nov 2022 09:43:36 +0000

We have about 20,000 cryptocurrencies, with over 9,000 of them actively trading. Similarly, we have over 4,000 exchanges, with over 500 of them being active. The CoinMarketCap website has all of this information. However, I wanted to be able to analyse cryptocurrency data beyond just how many cryptocurrencies are active. To accomplish it, I ended up leveraging AWS Serverless services.

This blog post describes how I used AWS serverless services to visualize cryptocurrency data using AWS Quicksight. So, if someone wants to do something similar, they know where to start. To achieve my goal, I used the following AWS services:

Lambda Functions
S3
AWS EventBridge
Step Function
QuickSight

Lambda Functions

In this use scenario, I have three lambda functions. Each lambda function runs a given function and saves the result to an S3 bucket. One of the lambda functions is shown below, which uses the Coinmarketcap map API and saves the response to an S3 bucket.

from os import path
import json, boto3, sys, uuid

# from botocore.vendored import requests

from requests import Request, Session
from requests.exceptions import ConnectionError, Timeout, TooManyRedirects

s3_client = boto3.client('s3')

def lambda_handler(event, context):
    url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/map"
    headers = {
        'Accepts': 'application/json',
        'X-CMC_PRO_API_KEY': 'xxxxxxxx',
    }

    session = Session()
    session.headers.update(headers)

    try:
        response = session.get(url)
        data=json.loads(response.text)
        json_body=json.dumps(data['data'])
        AWS_BUCKET_NAME='coinmarketcap-data'
        S3 = boto3.resource("s3")
        bucket_name = S3.Bucket(AWS_BUCKET_NAME)
        bucket_name.put_object(
            ContentType='application/json',
            Key="map-file.json",
            Body=json_body,
            )

        return {'statusCode': 200,'body': json.dumps('file is created')}
    except (ConnectionError, Timeout, TooManyRedirects) as e:
        print(e)

S3

Lambda functions write their data to an S3 bucket, data in S3 is then later consumed by Quicksight for visualization.

Here is an extract from the map data file stored in S3 bucket.

AWS Step Function

The step function is used to connect Lambda functions into a serverless workflow.

AWS EventBridge

For this specific use case, the EventBridge is used to schedule to run the Step function every 24 hours.

AWS QuickSight

Finally AWS QuickSight is used to load the data from S3 bucket and visualize it as shown below.

The whole AWS serverless architecture for this use case looks like this:

Thanks for reading!

If you enjoyed this article feel free to share it on social media 🙂

Say Hello on: Linkedin | Twitter | Polywork

Blogging: Dev | Hashnode

Github: hseera