DEV Community: Arthur Azrieli

How to Handle Alert Fatigue

Arthur Azrieli — Fri, 28 Mar 2025 13:42:51 +0000

A very important aspect for many developers and DevOps is handling alerts. An even more important and often overlooked aspect is alert fatigue. Alert fatigue is caused by a high volume of alerts of which many are false positives, related alerts, or duplicates. Alert handlers become so used to alerts that they disregard and miss important ones. It takes only one or two missed alerts to bring a system to a halt. However, alert fatigue is nothing but a symptom of the underlying issues that need to be addressed.‍

What we'll cover and learn about alert fatigue and Alertmanager

In this article we want to discuss factors that could lead to alert fatigue, how to identify them and how to handle them. In addition, we will introduce Prometheus Alertmanager and how to use it in order to handle the very same scenarios that lead to alert fatigue.

Prerequisites

Before diving in, there are a few prerequisites if you want to follow along with the article:
Prometheus installed - install by helm chart
Alertmanager installed - alertmanager is installed by default when installing Prometheus helm chart
Knowledge of yaml‍

Too Many Alerts

Developers and DevOps are busy people. When a significant part of their time is spent on looking into or muting alerts that don't matter, we are looking at alert fatigue. To handle this situation we need to ask ourselves why there are so many alerts and why so many of them are false positives, duplicates, or simply of no value.

Easy Trigger Alert

There's a tendency to sometimes overdo alert configuration, mainly as precaution. For example, we would like to know if a service is using a lot of memory or if its CPU is spiking so that we can react in time in case it develops into service disruption. In an attempt to have a preemptive encompassing view of the system, we sometimes configure too many alerts whose thresholds are unjustified. To justify an alert, we need to ask if the CPU or memory spike are really a concern. It's not uncommon for services to work a little harder at times. If the historical data shows spikes that resolve themselves and don't correspond to incidents, the alert should not be configured at all. Besides, If we are concerned with service disruption due to resource shortage we should consider automatic scaling, not setting more alerts.‍

Alert Configuration Cleanup

Not setting more alerts is one part of prevention as the best medicine. Dealing with the ones that are already set is the other. Alerts that are already set should be examined through historical and operational lenses:

Have alerts been triggered many times before?
Have they self-resolved?
Have they really indicated some system or service disruption?
Are they perhaps related to other more significant alerts?

These questions will help us discover alerts that can be removed to prevent alert fatigue.

However, for mature and perhaps more complicated systems with many services and moving parts, going over hundreds of configured alerts and determining whether they are important or redundant can prove to be quite difficult and time consuming. There's always the question of ROI when coming to clean up alerts. If we spend a sprint on cleaning alerts it might benefit us down the line by reducing alert fatigue, but we just missed a sprint where we could deliver features and bug fixes. So the trade off is always there. We'd still argue that alert cleanup should take place, even if in small increments. While we are cleaning them up bit by bit, we can introduce an alert manager whose purpose is to provide additional protection against alert fatigue.‍

Alert Manager to Handle and Prevent Alert Fatigue

An alert manager like Prometheus AlertManager provides a robust way to manage alerts. In the context of handling alert fatigue, the most significant aspect of an alert manager is its ability to group and inhibit alerts.‍

Correlating and grouping alerts

Let's look at an example of correlating and grouping alerts. Imagine a scenario in which a datastore such as a database, search engine, or queue manager is reporting high CPU and memory consumption. Services that depend on this datastore might experience difficulties communicating with it. They too might trigger an alert indicating that they cannot communicate with the datastore. With Prometheus AlertManager, it is possible to configure that if there's an alert for the datastore resource consumption all other related alerts will be grouped together and sent to the same receiver. This way we can see the underlying cause and which services are affected.

Given that alerts are properly labeled and configured to include the service name, team, cluster, region and any other attribute of significance, we can configure an alert as follows:

alertmanager.yaml:
global: {}
receivers:
  - name: 'default-receiver'
    webhook_configs:
      - url: 'http://example.com/webhook'
  - name: 'data-dev-receiver'
    webhook_configs:
      - url: 'http://example.com/webhook-data-dev'
route:
  group_interval: 5m
  group_wait: 10s
  receiver: default-receiver
  repeat_interval: 3h

  routes:
  - matchers:
      - team="data-dev"
    group_by: ['cluster', 'database']
    receiver: 'data-dev-receiver'
    continue: true

In the routes directive, we are saying that any alert meant for the data-dev team should be grouped under the data-dev-receiver by cluster and database. To simulate, we can run these commands:

curl -H 'Content-Type: application/json' \
     -d '[{
           "labels": {
             "alertname": "DatabaseUnreachable",
             "team": "data-dev",
             "service": "aggregator",
             "database": "analytics",
             "cluster": "prod",
             "severity": "critical"
           }
         }]' \
     http://localhost:9093/api/v2/alerts

curl -H 'Content-Type: application/json' \
     -d '[{
           "labels": {
             "alertname": "DatabaseResourceConsumptionHigh",
             "team": "data-dev",
             "database": "analytics",
             "cluster": "prod",
             "severity": "critical"
           }
         }]' \
     http://localhost:9093/api/v2/alerts

This is what it would look like in Alertmanager:

Inhibiting and Suppressing Alerts

In other cases we want to suppress related alerts rather than group them with the underlying alert. For example, a database has the following alerts configured:

Consuming too much resources
Unreachable
Connections about to max out
File descriptors limit is about to be reached
Queries take too long to execute
Many failed queries

Any of these alerts can be followed by the others. If an alert triggers for resource consumption, then an unreachable alert might also trigger. If queries take too long to execute, an alert for resource consumption might trigger as well.

So any alert might be followed by other alerts which will create a cascade of incoming alerts when all is needed is just the one. Instead of having them all triggered one after another, Prometheus Alertmanager can be configured to suppress a subset of alerts if a specific alert is active. To handle this situation, we can create the following inhibit rules:

inhibit_rules:

  - source_matchers:
      - alertname="DatabaseResourceConsumptionHigh"
    target_matchers:
      - alertname="DatabaseUnreachable"
    equal: ['database']

  - source_matchers:
      - alertname="DatabaseResourceConsumptionHigh"
    target_matchers:
      - alertname="DatabaseSlowQueries"
    equal: ['database']

  - source_matchers:
      - alertname="DatabaseResourceConsumptionHigh"
    target_matchers:
      - alertname="DatabaseFailedQueries"
    equal: ['database']

We're saying that if there's an active alert for DatabaseResourceConsumptionHigh in the specific database, any DatabaseSlowQueries or DatabaseFailedQueries alert will be inhibited. Inhibited means that the alert will not trigger but will still be visible on demand. The idea here is if there's an alert on resource consumption, we already know that there's something going on with the database. We don't need to be paged for all other issues as they are probably related yet we still have them at hand for investigation.

This is what it would look like in Alertmanager:

Inhibited alerts don't appear when Inhibited is not selected

Inhibited alerts appear when Inhibited is selected

Personally Contributing to Prevent Alert Fatigue

The ability to suppress alerts using Prometheus Alertmanager can also be used by the individual contributor to help prevent alert fatigue.‍

Taking Ownership of Alerts

An engineer who is making changes to the system and knows that alerts might trigger, should use the AlertManager ability to silence and inhibit alerts. It goes without saying how disturbing it is to get a high number of alerts in the middle of a workday, especially when these alerts are false positives and could have been silenced in advance.

If the alerts cannot be silenced because they are required as indicators during operational changes, the engineer should assume the PD or DOD shift. It might sound trivial but from personal experience we can testify that this is a common problem.‍

Handling Alert Fatigue is a Continuous Process

Using an AlertManager and personally assuming responsibility over alerts are the two determining factors in handling alert fatigue. These will greatly reduce the number of alerts and pagers to mitigate and prevent alert fatigue. However, these are but the tools and methods to achieve the goal. They will have to be constantly used and reimplemented.

It's important to remember that the effort to manage alerts and avoid alert fatigue is an endless process. As systems evolve their monitoring stack evolves as well. Currently configured alert handling rules might have been set up with oversight and should now be revisited. Think of it as a kind of monitoring for your monitoring. We have alerts and alerting rules in place but we have to always ask ourselves whether they are properly set and whether conditions have changed that merit revision of alerting rules.

From gatekeepers to service providers: reimagining DevOps relationship with developers

Arthur Azrieli — Fri, 28 Feb 2025 19:27:44 +0000

Most of us, if not all of us, are service providers. It doesn’t matter what position we are in. Each person in an organisation renders service onto the organisation, even the CEO and board of directors. Perhaps the only ones who are not service providers are the investors but perhaps they too render their service onto someone or something else. We begin with the notion of service providers because it is a crucial factor in the relationship between DevOps and developers.

The Relationship between DevOps and Developers

The relationship between DevOps and developers is as delicate and complicated as it is crucial for the whole organisation. Delicate because any friction causes setbacks in development. Complicated because it spans across many domains of knowledge and various requirements. And crucial for the whole organisation because if it’s not optimal, developers don’t deliver quality releases on time.

Luckily, the core of the relationship between DevOps and developers is technological, so handling challenges and adaptations is technological in nature. There are ways, purely technological ones, to reduce the pressure DevOps face on the one hand, and on the other hand remove obstacles from the developers’ way and let them become independent in their work.

DevOps Are Service Providers but on a Different Scale

DevOps is developer operations abbreviated, and it’s easy to see from this terminology that DevOps are the exclusive service providers for developers and should gear themselves accordingly. However, it appears that acting as service providers doesn’t come easy to DevOps due to the scale and complexity of what’s required from them as service providers. As a result, a lot of friction and dissonance exist between DevOps and developers in many organisations. To solve the friction and dissonance, we need to understand what type of service providers DevOps are, what’s the scale of their work, and how they can improve communication and operations to make their work, and the developers’ work, faster, easier, and more efficient.

Understanding the Role of DevOps as Service Providers

Most service providers usually provide their services within well-defined scopes. Developers develop, analysts analyze, QA verify, and so on. It’s true that these can also have their own internal customers like team members that require assistance. However, in such cases, the service or assistance is still within the scope and on a relatively small scale.

DevOps is a different story. Not only do DevOps have their own work cut out for them, they also support a lot of internal customers, and do so on a much larger scale and within a wider scope. DevOps assist RND people (and others such as analysts and sales engineers) in a wide variety of contexts:

Assisting in setting up local environments.
Granting roles and permissions to internal and external systems.
Troubleshooting issues with databases, microservices, and deployments.
Provisioning resources on demand.
Consulting on scale and security.

The problem here is that DevOps are a lot of times unprepared and unequipped for it. Neither in terms of understanding who the customer is, what they need, and how to give it to them with resources and restrictions in mind, nor in terms of how to do it at scale.

DevOps usually come into the job with a different mindset. A DevOps engineer probably sees themselves responsible for the development and maintenance of mainly the following:

Infra - Cloud, different environments.
Monitoring, logging, and alerting systems.
CI/CD infra.

Most DevOps will agree that it’s okay to add to this list ongoing support for developers and other stakeholders over a wide variety of contexts, systems, frameworks, and platforms.

However, it’s this very same wide variety of domains that DevOps support that prevents them from providing good service. When DevOps are unable to provide this service, it creates a lot of friction and dissonance between DevOps and developers.

The Dissonance Between DevOps and Developers

There’s no shortage of dissonance and conflict between DevOps and developers. Let’s look at some real-life examples.

What DevOps might say:

Developers don’t do the bare minimum to solve issues themselves before turning to DevOps.
Developers' requirements from DevOps are always the path of least resistance.
Developers don’t develop with security and scale in mind.

On the other hand, developers may want to counter-argue:

DevOps are not responsive enough, neither in terms of time to resolution nor deliverables.
DevOps impede development and velocity through requirements and bureaucracy.
DevOps don’t develop with developers in mind to assist them and facilitate their work.

It’s easy to see the dissonance when these complaints are put side by side and one after another. Putting this conflict into structure reveals the disconnect and distance between what one side needs and what the other side is capable of providing.

Since DevOps have their own duties to fulfill, adding to that extensive support adds pressure. This pressure could make DevOps compromise on the service they give because they are short on time and resources. So they expect developers to be self-sufficient and efficient when asking for support. What DevOps see as inefficiency in developers stems from the fact the developers are the most pressured entity in the organisation because they develop the product. When developers come across issues that prevent them from working they too are short on time and resources and need someone to assist them in a timely manner.

This is reality nowadays in many organisations. Both developers and DevOps are short on time and resources, and when the former approaches the latter, the latter needs to halt their work and assist, or development will suffer setbacks. The goal is to change this reality and realign DevOps and developers towards better collaboration.

Realigning DevOps with Developers

To realign DevOps with developers is not an easy task. The dissonance is not found in disagreement and differences. The dissonance is found in the fact that it’s not easy to change this reality and realign DevOps and developers towards better collaboration. It’s not enough to just tell DevOps that they are service providers and should do what they can to support developers in their day-to-day operations. It’s not enough to tell developers to approach DevOps like a customer requiring a service.

What’s lacking here is not verbal agreement but a well-defined, well-implemented framework of methodologies to help the two sides communicate and collaborate clearly and efficiently.

Moreover, the DevOps mindset must incorporate the idea that to provide services on a large scale, you need tools and you need to know what the customer needs, how and when. For DevOps, to provide services is to alleviate the pressure coming from developer needs and improve delivery. Let’s explore some ideas for improvement.

Communication Tools

Most companies use some form of ChatOps such as Slack or Teams. A dedicated channel for requests from developers is the first step. However, if there’s no structured way to submit requests for support or resources, it can become unmanageable and unwieldy really quickly. Many requests can come in at once and each of them might be related to something different.
To tackle this issue, it’s possible to install a request bot or request form in the dedicated channel. The request bot or form allows developers to submit requests in an orderly manner. It also allows DevOps to manage requests by queue and with more info and context to begin with. The form or bot should gather the following from the requester:

The nature of the problem - is it a request for support, a general question, or a request for resources?
What’s the environment in question - is it local, dev, or production?
The request itself - does the developer need to set up a new service, are they having issues with a service, do they need more permissions for internal and external systems?
If it’s a service - what is the name of the service and its dependencies (storage, docker repos, git repos, databases)?
If it’s a request for resources - quantitative measures such as CPU and memory and justification for adding resources.
If it’s more permissions - justification for permissions.
What steps were taken to try and troubleshoot - where applicable.
Any additional information that might be relevant - logs, metrics, documentation.

With the above in mind, now it’s time to see what can be automated or made self-serve to facilitate developer work by way of delegating and enabling.

Facilitating Developer Work

Most customers would prefer to do things themselves. Especially developers who always work in fast-paced environments and within time constraints. We’ve started with communication tools and now want to:

Gather info and analyze.
Discover areas that are constant pain points for developers.
Automate or delegate where possible.
Rinse and repeat.

This way DevOps can automate repetitive tasks such as granting permissions or provisioning resources. DevOps must also think about protecting their own customers and not be too permissive. Clear boundaries need to be defined when permissions or resources are asked because, like we said, DevOps are usually adamant when it comes to security and scale.

Beyond turning repetitive tasks to self-serve, DevOps should also strive towards making developers’ work as easy as possible. For a developer, an easier way to work could include:

Work - do everything on their own without needing anyone’s help.
Develop - disposable dev environments that are quick to set up.
CI/CD - easy to configure, easy to deploy, easy to revert.
Panic time - clean, well-scoped, logs and metrics that are easy to search.

At this point we are almost three quarters of the way in. Now all that’s left is to make sure that customers are well-aware of what’s at their disposal. DevOps can and should plan for building a body of knowledge to assist and educate developers on how to make the best of what’s offered to them by DevOps.

Summarize, Refine, Document, Educate

Once proper communication and procedures are in place to facilitate developer work, the body of knowledge should be assembled. Everything that doesn’t fall under automated requests and better troubleshooting and debugging tools should go into the body of knowledge.

The body of knowledge consists of the following:

Documentation
How-Tos
FAQs
Workshops

Composing and maintaining a body of knowledge is probably one of the most challenging things DevOps can do. It’s easy for DevOps to configure automations but most don’t know how to write in a clear concise manner. In addition, most customers don’t really bother to read the docs and if they do they just skim through. Even if developers do make use of documentation and workshops, knowledge is changing fast and there’s a need to adjust and update the body of knowledge accordingly. To tackle this challenge DevOps can encourage developers to take an active part in maintaining the body knowledge. An engaged customer is most likely a self-sufficient one. Perhaps the most important aspect of imparting knowledge is workshops. Not only is it easier to learn and understand by doing rather than reading, it also brings DevOps and developers closer together and strengthens their relationship.

DevOps as a Service: Maintaining Relationships at Scale

The first step towards improving the collaboration between DevOps and developers is understanding its scale. The scale is massive enough to put strain on the relationship, and like most scale issues, systems and procedures can be put in place and iteratively revised and improved to handle it.

DevOps must acknowledge that they are service providers at scale. Being service providers at scale, they must understand their resources, capabilities, and limitations in assisting developers, and put a system or procedure in place to handle it for them. DevOps must empower developers and delegate more responsibilities to them, therefore relieving pressure off of themselves while giving developers more agency.

Only when DevOps follow this set of principles will they be able to finally emerge as they have always meant to be: service providers at scale.

Proper mindset for handling data and databases: between scaling and failing

Arthur Azrieli — Fri, 28 Feb 2025 17:19:11 +0000

Startups and software companies put a lot of effort into what languages to use, what tech stacks to employ, and what cloud to deploy the app to but fail to put the same focus on their data and databases

Data is the core of the app and product from which everything is derived and upon which everything depends. Startups and software houses should put more thought into how they plan to gather, store, analyse, and use their data. Doing so could mark the difference between success and failure.

Avoid Common Data Architecture Regrets

Data is the raw material from which you mold your application. Anything else is just tools. If you treat your raw ingredients with proper care, the final recipe is bound to succeed.

The tried and true methods of optimizing data are abundant. It’s well known that databases can be tweaked to better perform through various means:

Indexing - speed up searches that rely on primary keys.
Normalization - keep data atomic and separate.
Query optimization - select only what you need, when you need it.
Partitioning - divide large tables into smaller ones.

So if it’s all tried and true and has been established, why do we highlight it as an overlooked part of many applications? That’s because there’s a long way to go from theory to practice. The list above only represents what can be done to optimize performance, not how and when to do so. Moreover, in a fast-paced environment of software development and especially in startups, proper planning for data is sometimes pushed aside in favor of rapid growth.

Plan Ahead for Your Data

We’ve mentioned several ways to optimize database performance, but what we should really focus on is planning for the data. Questions like what sort of data it will be, what will be the format, and what sort of manipulations it will undergo. Perhaps even more rudimentary than the type and usage of data is the database itself and how it fits your application.

Get to Know the Database

There are two main types of databases these days: relational (SQL) and document-based (NoSQL).

NoSQL:

If your application needs to handle single yet flexible documents.
If you predict large amounts of data that might be distributed and sharded.
If you expect a lot of unstructured data.

SQL:

If your application requires rigid, well-defined schemas and relations.
If your application requires consistency throughout the datascape.
If you intend to digest columnar data using big data tools.

Once you’ve chosen the database to work with, ask yourself again what your use case is. Inform yourself as to what others experienced working with MySQL, MariaDB, PostgreSQL, MongoDB, to name a few. Find the setbacks that others faced and see if at any point in the future you might face something similar.

Get to Know the Data and its Characteristics

The way you design your data now will impact you in the future. It’s a hard task, but force yourself to think of what other functionality you have in store and plan to implement. See if the current data scheme and models allow easy integration of such functionality.

Functionality implies data moving around and being updated constantly. Consider the behavior of your data:

If you do a lot of writes but fewer reads, opt for throughput.
If you do many reads but fewer writes, opt for io and use caching.

Load and Stress the Data

Data-related performance issues mostly hit you when you least expect them. The smooth functionality that you are used to is not attributed to the choice of database or data scheme. It’s mostly attributed to lack of load and stress on the database. This load and stress is what you should strive for.

Again a hard task ahead that requires you to accept that tens of thousands of requests per minute can easily become millions. It’s easy to list and discuss ways to optimize data manipulation and retrieval, but no one gains experience and knowledge without trying.

If you want to know if your data is well-structured and well-retrieved:

Try to write and read more than you imagine would be possible.
Only when it stops working do we look under the hood to find and fix the problem.

Like we said earlier, anything else is just tools. The data is the heart and core of the application and should be created like one: consistent, resilient, scalable.

Protect Your Data

With the efforts of choosing a database, data models, and optimizing their usage behind you, you should think about protecting your data.

Protecting your data from bad actors goes without saying. It’s the internal actors that you need to shield the data from. Internal actors can be services and humans, and since humans make mistakes, so do services.

Consider the following means of mitigating accidental service disruption or, worse yet, data loss or corruption:

Back up the data and plan for deploying from a snapshot.
Limit access from the get-go.
Reads from replicas, no human ever writes directly to the data.
If a service is the owner or main user of a table or database, other services request data through internal APIs.
Monitor the database CPU and memory and plan ahead in case you need to scale.
Look for, kill, and find the source of long-running queries to detect misbehaving services.

Keep The Data Clean

It’s not enough to protect your data. You also have to keep it nice and tidy. A lot of data accumulates through the product lifecycle and more often than not becomes stale.

Modern hard drives are fast, reliable, and reach terabytes in volume, but that doesn’t mean you should fill them with data.

Too much data puts strain on the disk and memory, not to mention that more data means longer queries, even with indexing.

Consider the following as ways to keep your data clean:

Don’t do soft deletes.
Scan and find least retrieved data and archive it.
If you’re using PostgreSQL, use vacuum. If you use MySQL, use optimize. Do the same for any other database you use.
Be wary of making changes – don’t add tables that duplicate data.

Keep Your Data in Mind

Out of all the aspects and methods we discussed, there’s one conclusion to be drawn:

Data is the most important, most overlooked aspect of software development.

To keep your data in mind means to consider all the pros and cons of choosing a database.

To keep your data in mind means you always check how data retrieval affects performance.

To keep your data in mind is to follow these principles:

Choose the right database for the workload.
Create indices and optimize queries.
Optimize I/O through hardware adjustments and caching.
Get rid of unnecessary data – no soft deletes.
Check and check again that high volume doesn’t create bottlenecks.
Back up your data and limit access.

Strive to apply the principles listed above because no matter what you do with your app or product, it’s almost always related to data.

Always keep in mind that Data is the foundation. When it’s well-maintained, the whole system benefits.

Practical Tips for Kubernetes Upgrades for Startups

Arthur Azrieli — Mon, 10 Feb 2025 18:58:35 +0000

Upgrade Kubernetes with confidence: A step-by-step guide to ensure seamless updates, maintain stability, and avoid breaking changes.

The all-too-popular Kubernetes upgrade-storm

There comes a day when you get a notification that the current Kubernetes version that you are running is reaching its end of life. Best case scenario you open a ticket knowing full well that this ticket will either be pushed down the list of priorities or be forgotten completely. After all, you have other priorities such as releases and bug fixing. You are a fast-running startup that needs to bring in new business in order to grow. Upgrading Kubernetes is the least of your concerns right now.

You will have to upgrade eventually

But the day finally comes when you need to upgrade and one of the following could be the trigger:

Your Kubernetes version actually finally reached its end of life.
RND management finally decided it was time to upgrade.
You need to upgrade regardless of end of life because some critical components in your cluster need upgrading for a bug fix or a feature that you need.

You look up the helm chart or operator that is running in your cluster and realize that you cannot upgrade to the newer versions because they are incompatible with your current Kubernetes version. So you need to upgrade the cluster in order to upgrade the helm charts. And to top it all off, it has been decided to upgrade Kubernetes all the way to the latest versions and you find yourself needing to upgrade 4 versions forward.

Every Kubernetes upgrade has the potential to introduce breaking changes. In most cases it’s deprecated APIs or APIs that moved from one API group to another. This will affect anything in your cluster that relies on these Kubernetes APIs. Now take this and do it four times. You need to carefully plan how to approach and execute the upgrade:

Scope and price the process in terms of effort and time to completion.
Create an upgrade plan and iterate over it by testing on lower environments.
Set a maintenance window.
Declare code freeze.
Upgrade and verify.

This is a challenging process that will exhaust you. It is labor intensive and very error prone. If you upgrade a library in some microservice you can test it locally, the scope is almost always isolated to this specific micro service and in any case the blast radius is relatively small. But when you upgrade a Kubernetes cluster you are upgrading the entire system and anything going wrong could have serious consequences.

How to approach the upgrade

Before discussing ways to approach an upcoming upgrade we need to address the elephant in the room. Once you need to upgrade, you’re probably short on time, short on resources, and need to upgrade several versions forward while making sure that the app stack itself and everything else that runs on your Kubernetes cluster remains functional. That is not the way to go.

What we derive from this situation is the first principle of how to approach the upgrade - upgrade small, upgrade continuous. Once we realize and implement this principle we can move on to what you need to do in order to successfully upgrade your Kubernetes cluster.

Upgrade small, upgrade continuous.

Once we realize and implement this principle, we can move on to what you need to do in order to successfully upgrade your Kubernetes cluster.

Upgrade small upgrade continuous

Remember the day when you got the notification that your Kubernetes cluster versions reached its end of life? Well this is the day where you waste no time and put this task in the sprint.

Opponents of this approach might say that a startup cannot afford to jump on every upgrade because there’s more pressing business to conduct. But a startup also cannot afford system instability. The longer the wait the more unstable the system might become and the upgrade will be harder and harder, especially if more than one upgrade is in question. So upgrade small, upgrade continuous. This goes for components in the cluster as much as the cluster itself.

Keeping your helm charts, operators and controllers up to date will almost always guarantee that you will not have to upgrade them when upgrading your Kubernetes cluster. There is no doubt that a startup should think first how to bring in money. If the Kubernetes cluster upgrade competes with a feature that will bring in new business, the feature will almost always win.

However, by insisting on upgrading small and keeping up to date, you provide yourself with breathing room for when a feature or a bug fix are really critical. You can allow yourself to skip the upgrade and focus on business because the end of life of your Kubernetes cluster version is farther down the road.

Test and verify on lower environments

Another principle worth discussing is testing and verifying the upgrade on lower environments. The term lower environments obviously means dev and stg but in many cases dev and stg environment represent the app stack and less so the infrastructure.

This means that when testing on lower environments they have to be identical to the production cluster that you are about to upgrade. As identical non-critical environments, you can allow yourself to make mistakes which are the best way to learn.

Upgrading Kubernetes is a difficult task. Having the ability to try it out without fear of service disruption is liberating and will allow you to experiment more therefore better preparing yourself for the upgrade day.

When you eventually test on lower environments, don’t just upgrade and settle for a working cluster. Remeber that lower environments are meant to represent the architecture and app stack of higher environments. Consider the following when upgrading consider the following:

Monitor the environments through metrics and logs to check for anything suspicious or out of the ordinary:
- A critical component fails to be scheduled - pods in crash CrashLoopBackOff, pods failing to satisfy liveness and readiness probe, nodes don’t scale when the cluster is loaded, etc.
- kube-proxy is not alive and well and services cannot talk to each other.
- kube-dns is not alive and well and services fail to resolve host names.
Run e2e tests on your app stack. Verifying that your app stack functions as it should in an upgraded environments will give extra confidence that the upgrade is going well:
- Run e2e tests.
- Run integration tests.
- Run load tests to verify that deployments scale accordingly.

There’s a caveat though in this approach that we need to address. You have to provision and maintain these environments which means more resources allocated to the upgrade process even if it’s not in the pipeline. But there’s no better approach. It’s a “measure twice cut once” and “invest money not time” rolled into one.

Try out the upgrade first and then execute and do it on preallocated environments. The stability you will achieve will contribute to the overall health of the system and the organization itself. No amount of money can compensate for dev teams overworked by system instability. It could also prove a source of churn where instability drives away clients and even prospects.

Now it’s time to upgrade

Let’s assume that we are in an ideal world where you have your lower environments ready and well-maintained and you have allocated time and resources for continuous upgrades. How do you prepare for an upgrade? There are several things you need to do.

First of all, you have to thoroughly read the release notes. And it doesn’t mean scrolling through them but reading them line by line. It’s a time consuming task but it follows the principle of “measure twice cut once”. A lot of what you will read won’t be relevant. A lot of what you will read will be invaluable. Dedicate time and patience to this task. Tutorials and guides are obviously welcome but try to remember that not all environments are alike.

Now that you have a sense of what’s heading your way in terms of the effort put into research, you could use an automated process to give you a head start. You can find exactly that in kubepug which is an open-source Kubernetes pre-upgrade checker. What you can and should do with kubepug:

Run kubepug against your current Kubernetes cluster version to get the following:
- A list of deprecated APIs.
- Any objects affected by API changes.
- What APIs should be used instead of deprecated ones.
If all goes well and we wish you that it will, run kubepug once more because it is also capable of verifying current versions.
If all goes well and we wish you that it will, run kubepug once more because it is also capable of verifying current versions.
Trust kubepug but verify that everything that it drew you attention to was indeed upgraded or replaced and that it’s consistent with the release notes.

Once you gather the information and mode of operation from release notes and guides, go look at your Kubernetes cluster and find everything that both exists in your cluster and is referenced in the information you gathered. The match between the two is the basis for your upgrade plan.

Automate and summarize

We’ve mentioned a few times that the upgrade process, especially everything that precedes it, is very arduous and time consuming. This is where you automate the discovery and summary process.

Use LLMs to summarize and highlight information that you gathered and other tools to scan, analyze and inform you on changes between versions. Another aspect of the upgrade process is to compile, document and implement the upgrade process itself. Laying down the foundations of upgrading small and continuously is perhaps the most important aspect even more than the upgrade itself.

It’s true that the goal is the eventual Kubernetes cluster upgrade, but how it’s carried out will determine the measure of peace of mind that you will have when approaching this important task.

Yours is a startup and should start well

Kubernetes is one of the best things to have happened to the tech industry. By using it, your startup avoids the pain of having to provision your own orchestrator. Take into account that the time you save for using Kubernetes rather than maintaining your own solution, is time to invest in paying respects back to Kubernetes.

And to do that you need to consider that for Kubernetes to continue serving you it needs to be up to date and well-maintained. Then and only then will it guarantee the highest level of stability. And a stable infra for a startup is priceless as it allows you to grow and rarely holds you back.

So for all your successful upgrades to come, adopt the mindset that we are trying to convey.

Give the Kubernetes upgrade its place in the development pipeline. Like we highlighted, an upgraded, well-maintained cluster is an invaluable resource. Small to medium effort every now and then is better than an out-of-the-blue urgent upgrade.

Be ahead of the upgrade. Don’t wait for it to come to you. Seek it proactively and open a ticket with a due date. Add yourself a calendar reminder. Allow yourself time to educate and prepare yourself. Yes there are automated tools like kubepug that we mentioned but we need to know how to use these tools and rely on them to the extent that they don’t have the final say.

Test on lower environments and verify and validate by looking at metrics and logs. Validate further by making sure that the app stack functions as it should.

These principles don't guarantee smooth upgrades as the unexpected is almost always bound to happen. However, they do guarantee successful upgrades that will instill in you greater confidence for the current upgrade as well as future upgrade. You’re a startup and adopting these principles and mindset will prove itself not only when upgrading your cluster, but in anything that you set out your startup to become and achieve.

Deploy a Kubernetes App & AWS Resources using Crossplane on Kubernetes: Part 2

Arthur Azrieli — Thu, 31 Oct 2024 12:19:56 +0000

To properly enjoy this article

This tutorial assumes you already followed the steps in part 1: Deploy AWS Resources using Crossplane on Kubernetes.

Also, this is the Github Repository we’ll be using:
https://github.com/MeteorOps/crossplane-aws-provider-bootstrap

What’s this article about?

In this article, we’ll cover a use-case that can benefit from Crossplane: full environment deployment.

This is a step-by-step guide with an example and a Git repository, so by the end of it, you should be able to deploy a sample env.

You can technically walkthrough the entire thing by "copy-paste" and everything should work. But, diving into the explanations with an extra 5–10 minutes will leave you with longer-term value.

Hope you enjoy!

What to expect from this article?

By the end of it, you’ll understand:

How Crossplane can be used for full environment deployment
How to deploy a sample app with AWS resources

What not to expect?

This article guides you through a simple application deployment, and not a full set of apps.

It also doesn’t go into using Crossplane in conjunction with Helm, but does cover important principles regarding it.

Why Crossplane for the Full Environment Use-Case?

When you want to deploy a full environment, it usually involves 3 layers:

Infrastructure: Resources the application needs to run well
Application: The programs built by the company to serve users
Data: The data the application uses

But you already know that.

The thing is a tradition developed, and Crossplane sort of broke this tradition.

The tradition was this process: Build infrastructure, Deploy application on top.
How did Crossplane break this tradition?
The application deployment can now provision infrastructure required by the application.

Pull-Request Environments are also easier

By creating a namespace with all of the apps and the AWS resources required with Crossplane, the use-case of creating a full environment per Pull-Request as part of the CI becomes much easier.

That's a nice benefit of such setup for companies utilizing the feature-branch or Gitflow approaches.

A Traditional Full Env Example

To provision and deploy a full environment in the past, the process would generally look something like this:

Provision VPC+EKS+... using Terraform
Use Terraform to bootstrap the cluster with a CD tool (e.g., ArgoCD)
ArgoCD looks at a repo that deploys all apps from there
An application needs a new S3 Bucket, so the developer writes Terraform code for it
The application gets removed after a while (but the bucket stays)
Someone needs to remember that bucket was owned by that app and remove it from Terraform

A Crossplane Full Env Example

To provision and deploy a full environment with Crossplane the process is similar (we still need a Kubernetes Cluster to start with for the initial environment):

Provision VPC+EKS+... using Terraform
Deploy Crossplane’s prerequisites to the cluster with Terraform
Add Crossplane resources to application Helm Charts (so they get their required infra upon deployment)
Create a Crossplane manifest to deploy the Helm Charts + Some shared infra required by all apps
When an application is removed, its AWS resources are gone with it
When an entire environment is terminated, its AWS resources are gone with it

Crossplane in Helm vs. Helm in Crossplane

When using Crossplane alongside Helm, the question arises:

Should Helm apply the Crossplane code? Or, should Crossplane apply the Helm Charts?
I'm glad you asked it - the answer is both, depends when.

Reasons for Crossplane in Helm:

Create or modify app-specific resources when that app is deployed
Delete app-specific resources when that app is deleted

Reasons for Helm in Crossplane:

Manage dependencies between resources and applications using Crossplane
Create shared resources that are not owned by a single application

The Step-by-Step Guide

Deploy the simple application alongside a S3 bucket using a Crossplane Composite Application.

Before proceeding

Make sure you follow the steps in the 1st article (takes 3-minutes to just copy-paste the code snippets into your terminal and run the entire thing).

Deploy the Crossplane Kubernetes Provider

Prepare the AWS Credentials for the Application to be able to use AWS
Run the following oneliner to create the Secret containing the AWS credentials in the right format as required by the Application (the application will simply run aws s3 ls to show the bucket):

kubectl create secret generic aws-creds \
--from-literal=aws_access_key_id=$(grep -i aws_access_key_id creds | awk -F' = ' '{print $2}') \
--from-literal=aws_secret_access_key=$(grep -i aws_secret_access_key creds | awk -F' = ' '{print $2}')

Make sure it was created as expected by fetching the secret:

Deploy the Crossplane Kubernetes Provider resources using the k8s-provider-bootstrap.yaml file

kubectl apply -f k8s-provider-bootstrap.yaml

Make sure the provider was created and is ready before proceeding to the next steps:

kubectl get providers provider-kubernetes

You should see something like this:

Deploy the Crossplane Kubernetes Provider Configuration using the k8s-provider-conf.yaml file:

kubectl apply -f k8s-provider-conf.yaml

This is done separately as it needs to happen after the Provider resources were created.

This is where we tell the Crossplane Kubernetes Provider in which Kubernetes cluster it should operate when it’s creating resources.

Create a deployable unit of an App & AWS Resources using Crossplane

Here we do 3 things with 3 files:

The composite-app-xrd file:
Contains the CompositeResourceDefinition (XRD) for the K8sApplication by using the Composition of a K8s Deployment and S3 Bucket (described below)
The composite-app-composition file:
Contains the Composition definition which creates both the Kubernetes Deployment and the S3 Bucket
The composite-app-example file:
Calls the CompositeResource defined by composite-app-xrd file

Crossplane Resources Files Breakdown & Creation

composite-app-xrd.yaml

~ K8sApplication CompositeResourceDefinition
This defines a composite resource for a Kubernetes application, with bucketName and bucketRegion fields in the spec. Users can claim this resource as K8sApplication.

The K8sApplication CompositeResource (XRD) accepts the bucketName & bucketRegion fields and uses them to create an S3 Bucket, and to create a K8s Deployment of a mock “service” that simply runs aws s3 ls to see the bucket.

~ Deploy the CompositeResourceDefinition (XRD)

kubectl apply -f composite-app-xrd.yaml

composite-app-composition.yaml

Defines a Composition of resources that can be created by a CompositeResource.

This is where we define the Composition that creates a combo of a Kubernetes Deployment with the mock “service” that runs aws s3 ls as well as the S3 bucket — The CompositeResource simply calls this resource.

~ Deploy the Composition

kubectl apply -f composite-app-composition.yaml

composite-app-example.yaml

Deploys the actual K8sApplication CompositeResource, and passes the details of the region in which the bucket should be created, and the name of the bucket (both are also passed to the Kubernetes Deployment as environment variables that helps it access the same bucket).

As mentioned above, the CompositeResource calls the Composition which creates the resources using the Crossplane providers.

Deploy the app by running the following command:

kubectl apply -f composite-app-example.yaml

Look at your pretty Application

Fetch the K8sApplication resource you’ve just created by running the below command obsessively until it’s marked as Healthy:

kubectl get K8sApplication

You should get something like this:

Print the logs of the application and see it fetching the AWS S3 Bucket:

kubectl logs -l app=awscli
# 2024-10-17 16:00:31 my-app-bucket-nqzhx-xzjcq
# 2024-10-17 16:00:50 my-app-bucket-nqzhx-xzjcq

Cleanup

kubectl delete -f composite-app-example.yaml

Recap

To briefly recap what you did here:

Prepared Crossplane for deploying a mix of Kubernetes and AWS resources
Defined the manifests required to deploy an app built of a Deployment and a S3 Bucket
Sharpened your grasp on some Crossplane concepts
Discussed some use-cases for which it’s useful

Hope you enjoyed this article, and if you are interested in another article about something related (or unrelated), please convince Michael it’s a good idea at michael@meteorops.com.

Disclaimer: In actual environments or production, it’s essential to fine-tune the permissions in the different manifests. Instead of using access keys and secret keys directly, consider implementing IAM Roles for Service Accounts (IRSA) to manage permissions more securely.

Deploy AWS Resources using Crossplane on Kubernetes

Arthur Azrieli — Wed, 18 Sep 2024 23:50:11 +0000

In this article we will be talking about Crossplane as an Infrastructure as Code (IaC) tool that is running on Kubernetes, why should we use it and how you configure AWS provider to start creating resources, we will be going through a full step by step example for you to be able to create your first resource with Crossplane

Who is this article for?

DevOps engineers interested in learning another IaC tool
Developers that want to take more Ops responsibility and provision their own infrastructure
Engineering managers that are looking to implement an IaC tool in their company/startup

Why am I writing this?

I had some discussions with engineers that had some trouble to get started with Crossplane, it may be a little less straightforward than a well established tool like Terraform, some documentation isn’t precise for different use cases and providers and even ChatGPT’s code doesn’t seem to work at times. And here I am saving the day to make your life easier by giving you a step by step guide where you install and configure everything and deploy your first AWS resource using Crossplane.

Why should you even use Crossplane then?

There are certain use cases where Crossplane provides very powerful capabilities being able to create both applications and cloud resources, those can be used for ephemeral environments for example or for having a SaaS company provide full environments that could be self created by a tenant. Those environments could be created by just applying a Kubernetes manifest which is much simpler than starting to run traditional IaC plan and apply commands.

How this article works

We prepared a repository with resources to deploy everything needed.
We explain on each resource what it does.
We walk you through how to deploy Crossplane.
We deploy a S3 Bucket to make sure everything works.

Prerequisites

1. Clone the repository and step into it

git clone https://github.com/MeteorOps/crossplane-aws-provider-bootstrap.git cd crossplane-aws-provider-bootstrap

2. Make sure you have the required CLIs:

Install the AWS CLI & Authenticate it with your AWS Account
Install the Kubectl CLI
Install the Helm CLI
An existing Kubernetes cluster (we’ll be using kind)

Optional: Start a local kind cluster

brew install kind

kind create cluster

open /Applications/Docker.app

kubectl cluster-info --context kind-kind

Repository Overview

Link to the Github Repository: https://github.com/MeteorOps/crossplane-aws-provider-bootstrap.git

creds file‍ AWS credentials - should be filled with your own AWS credentials
crossplane-provider-conf file ‍Uses the creds file to create a Crossplane ProviderConfig (separated into a different file because it takes time for this resource to be ready)
crossplane-provider-bootstrap file ‍Creates the Crossplane AWS Provider, which enables creating AWS resources using Crossplane (and its dependencies):ServiceAccount, DeploymentRuntimeConfig, Provider, ClusterRole & ClusterRoleBindings
bucket-definitions & bucket-crd files ‍The Kubernetes Crossplane manifests that create a CompositeResourceDefinition and a Composition resource, which together define how to create a S3 Bucket (like a Terraform Module would). Note: The ‘Composition’ resource relies on the ‘CompositeResourceDefinition’.
bucket-example file ‍The Kubernetes Crossplane manifest we’ll apply at the end to create a S3 bucket using Crossplane

Deploy Crossplane

1. Fill the creds file with your AWS access keys
Get your AWS IAM User (not an SSO user as it requires a token to work) access keys and fill them in the credentials file

NOTE: for production usage, please create a Crossplane IAM user and use its access keys, or preferably use something like IRSA

2. Deploy the Crossplane Helm Chart
Add the Helm repository from which the Crossplane Helm Charts will be fetched

helm repo add crossplane-stable https://charts.crossplane.io/stable

Deploy Crossplane on your Kubernetes cluster in a new namespace named crossplane-system

helm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace

3. Examine your Crossplane Deployment
‍

Run the following command to get Crossplane's pods:

kubectl get pods -n crossplane-system

Then, you should see 2 pods: crossplane & crossplane-rbac-manager

4. Provide Crossplane AWS access by creating a Kubernetes Secret
Insert your AWS credentials to the creds file and run the following from the same folder:

kubectl create secret generic aws-credentials -n crossplane-system --from-file=creds=./creds

Make sure the secret was created as expected:

Run the following command:

kubectl get secret aws-credentials -n crossplane-system

You should see the aws-credentials secret:

5. Deploy the Crossplane AWS Provider
‍Creating a Crossplane AWS Provider requires creating a bunch of resources: ServiceAccount, DeploymentRuntimeConfig, Provider, ClusterRole & ClusterRoleBindings, and ProviderConfig
We divided the resources creation into 2 phases:

1 - crossplane-provider-bootstrap.yaml:
ServiceAccount, DeploymentRuntimeConfig, Provider, ClusterRole & ClusterRoleBindings

2- crossplane-provider-conf.yaml:
ProviderConfig

The reason for dividing it into 2 phases is that the creation of the ProviderConfig fails if we attempt to create it before the first set of Provider resources and dependencies is ready

Create the Provider Kubernetes resources using the bootstrap YAML file:

Run the following command:

kubectl apply -f crossplane-provider-bootstrap.yaml

Validate the creation readiness of the Provider & wait for it to be ready:

Run the following command to see the AWS Provider resource:

kubectl get provider

You should see something like this:

It might take 1-2 minutes to become Healthy.

Create the ProviderConfig resource & Validate its creation:

Run the following command:

kubectl apply -f crossplane-provider-conf.yaml && kubectl get providerconfig

Create a S3 Bucket using Crossplane

Create the CompositeResourceDefinition to define a S3 Bucket:

Run the following command:

kubectl apply -f bucket-definitions.yaml

Create the Composition to define a S3 Bucket:

Run the following command:

kubectl apply -f bucket-crd.yaml

Create the S3 Bucket Crossplane resource in Kubernetes:

kubectl apply -f bucket-example.yaml

When we installed the AWS Provider, it was installed with some Crossplane CRDs of the AWS Provider.
One of those CRDs is ‘bucket’.

Now we can check if the bucket was created by running kubectl get bucket against our Kubernetes cluster

Check if the S3 Bucket was created in AWS:

‍List you AWS S3 buckets and search for the newly created one:

aws s3 ls

Teardown & Cleanup

We’ll start by deleting the S3 Bucket Crossplane resource in Kubernetes, which will end up deleting the resource in AWS.
Eventually, if we used kind to spin up a local Kubernetes cluster, we’ll terminate the cluster to keep our workstation nice and clean.

Delete the S3 Bucket resource:

kubectl delete -f bucket-example.yaml

If used kind delete the cluster:

kind delete cluster

Useful Debugging Commands

kubectl get provider


kubectl logs -n crossplane-system deploy/crossplane -c crossplane


kubectl logs -n crossplane-system -l pkg.crossplane.io/provider=provider-aws

Terraform Starter Boilerplate for GCP using Terragrunt

Arthur Azrieli — Fri, 26 Apr 2024 11:16:24 +0000

The Boilerplate Github Repositories (stars are welcome ⭐)

What you deploy: https://github.com/MeteorOps/terragrunt-gcp-projects
Modules you can use: https://github.com/MeteorOps/terraform-gcp-modules ‍

Terraform mistakes that made me build this boilerplate

I built this boilerplate for a reason.

I saw what companies regret with how they implemented Terraform:

Writing all of the Terraform code in one main.tf file
Copy-pasting resources manually
Copy-pasting configuration throughout the codebase
No state separation and environment awareness

This is why they regretted the above:

One terraform apply could ruin an entire environment
Resources modifications required changes in multiple locations
Configuration modifications required changes in multiple locations
Accidentally deploying the wrong resources to the wrong environment

And so, I built this boilerplate for our clients (and you) to minimize regrets.

The focus of this boilerplate is managing GCP resources.

Is this boilerplate for you?

If you're a CTO, a DevOps lead embarking on a new project on GCP, or simply in search of a template to organize your Terraform repositories, this project is for you. Finding a well-structured example for deploying GCP resources can be challenging. My own search for such a template was unfruitful, leading me to develop a solution that I've decided to share openly and discuss in this article.

What should you expect from this guide?

By the conclusion of this guide, you'll have a thorough understanding of how to establish Terraform repositories using a best-practice folder structure for provisioning GCP resources. You'll also be equipped to execute a straightforward demo, witnessing an end-to-end workflow in action.

What shouldn't you expect from this guide?

An exhaustive library of modules for every resource in GCP. We kept the boilerplate minimal, so that you can utilize it for your needs.
You can fairly easily utilize existing modules you created or found using the boilerplate.

Getting Started

To begin, clone the essential repositories:‍

Primary Repository
Clone the terragrunt-gcp-projects repository to get started.

git clone git@github.com:MeteorOps/terragrunt-gcp-projects.git

Modules Repository (Optional)

For the modules used, clone the terraform-gcp-modules repository.

git clone git@github.com:MeteorOps/terraform-gcp-modules.git

Repository Structure Explained

The code organization follows a logical hierarchy to facilitate multiple projects, regions or environments.This structure gives you a number of benefits:

Hierarchical configuration: The configuration at each level cascades through the folders under it
State separation: The terraform state is saved per folder in a different path in a bucket, limiting the impact radius of changes
Dynamic-level of deployment: The deeper into the folder you go, the more specific resources you affect with one deployment

project
└ _global
└ region
   └ _global
   └ environment
      └ resource

Creating and using root (project) level variables

When dealing with multiple GCP projects or regions, passing common variables to modules can become repetitive. To avoid duplicating variables across each terragrunt.hcl file, leverage root terragrunt.hcl inputs to inherit variables seamlessly across regions and environments.

Deploy Using Terragrunt

Prerequisites

Install Terraform version 0.12.6 or newer and Terragrunt version v0.25.1 or newer.
Fill in your GCP Project ID in my-project/project.hcl.
Make sure the gcloud CLI is installed and you are authenticated, otherwise run gcloud auth login.

Module Deployment

To deploy a single module:

cd into the module's folder (e.g. cd my-project/us-central1/rnd-1/vpc).
Run terragrunt plan to see the changes you are about to apply.
If the plan looks good, run terragrunt apply.

Environment Deployment

To deploy all modules within an environment:

cd into the environment folder (e.g. cd my-project/us-central1/rnd-1).
Run terragrunt run-all plan to see all the changes you are about to apply.
If the plan looks good, run terragrunt run-all apply.

Testing Deployed Infrastructure

Post-deployment, modules will output relevant information. For instance the IP of a deployed application:

Outputs:

ip = "35.240.219.84"

A minute or two after the deployment finishes, you should be able to test the ip output in your browser or with curl:

curl 35.240.219.84

# Output: Let MeteorOps know if this boilerplate needs any improvement!

Clean-Up Process

To remove all deployed modules within an environment:

cd into the environment folder (e.g. cd my-project/us-central1/rnd-1).
Run terragrunt run-all plan -destroy to see all the destroy changes you're about to apply.
If the plan looks good, run terragrunt run-all destroy. ‍

Conclusion

This guide walks you through leveraging best practices for setting up and managing Terraform repositories for GCP with Terragrunt. These methodologies are designed to be straightforward, efficient, and easily adaptable to future projects or company needs.

P.S.

Feel free to subscribe to our Newsletter and learn about other insights and resources we release 👈🏼