DEV Community: jameslaneovermind

Of course monolith infrastructure is cheaper than serverless

jameslaneovermind — Wed, 09 Aug 2023 14:38:19 +0000

In recent years, teams have been buzzing about microservices, with many organisations jumping on the bandwagon. Even the US Air Force now runs its latest fighter jets on k8s. However, just like Agile, SCRUM, or the ‘latest’ software development methodology, success isn’t guaranteed. What we are seeing is a realisation that the complexity of Kubernetes has a cost. A cost that is not always beneficial unless running at a larger, more complex scale or team topology. This is why some teams are now making a reversal, returning to the monolithic architecture they once left behind.

Splitting applications up with APIs gives us a defined separation of responsibility. It’s hard for 100+ people to cooperate together to build a monolith application. But if you had 10+ teams of 10 people deploying their own microservices it’s easier to decouple and to deliver at the pace each team needs.

The problem…
Getting everyone to agree on what these individual services should look like is where problems arise. Do you assign a team to a single function, or is it based on business unit requirements? For APIs, who is deciding the definitions and are they being documented? Conway’s Law states that the design of a system mirrors the structure of the organisation responsible for creating it. While microservices can offer better separation between teams, this advantage may not always be realised due to the inherent team structure or even culture of an organisation. In such a situation, monolithic architecture may start to look more attractive.

So it’s not surprising to anyone that articles like Amazon Prime Video’s “Microservices to monoliths” will emerge from time to time. However, in this example, they needed to handle multiple state transitions per second as part of video streaming data. That’s really not a great match for serverless and led to some impressive cost savings for Amazon. However some question remain:

Isn’t this something that should have been made apparent in the upfront design?
Was this an example of jumping in headfirst and developing without thinking through the problem? Analysis paralysis is a real thing organisations face but could they of over-corrected on that a bit.
Or would you argue employing serverless technology for rapid product testing was a smart initial move? There’s value in just getting things out the door and iterating, but it seemed that could be happening with less and less foresight, leading to bigger issues and larger refactors/iterations.
In this example, the issue arose when they failed to recognise the expenses associated with step transitions, which then led to the subsequent optimisation step of transitioning from a step function to a single EC2 component.

With that being said, the original post from Prime Video Tech contains numerous gaps, leading to confusion and a seemingly inaccurate title of “From distributed microservices to a monolith application”. The process appears to be more of a refactoring rather than a complete transformation.

Deciphering the Monolith Puzzle
So where does that leave us? Choosing the right architecture for your organisation is a balancing act. It’s possible to maintain the separation of concerns and scale different APIs using a monolithic architecture while still enjoying the benefits of microservices.

To decide whether you need to move back to a monolithic architecture or fix issues in a distributed monolith consider:

The trade-offs and timeframe.
Analyse the pain and productivity loss from microservices over time and weigh it against the cost of migrating to a monolith. Taking into account factors like Conway’s law, team size/ topology, experience, and expertise.

Team Topologies — by Matthew Skelton and Manuel Pais does an excellent job at providing a framework (grounded in Conway’s Law) for structuring teams to meet the needs of users and align with the architecture of the systems you’re building.‍

Overmind is a SaaS Terraform impact analysis tool. It discovers your AWS infrastructure so that it can calculate the blast radius of an application change, including resources managed outside of Terraform. Helping you to identify the causes of outages by showing you which changes caused which problems. While also helping you to deploy changes faster by giving an impact analysis report before any change is made. From this report you can understand if the change can be confidently made, or held back if it’s too risky, preventing outages in the first place.

Solving Meta's top 4 outage causes: 1/4 Unexpected Dependencies

jameslaneovermind — Mon, 07 Aug 2023 15:18:23 +0000

In 2022 Francois Richard deliver an excellent talk at SRECon EMEA about how Meta drained every backbone router simultaneously (you remember, it was that outage). Here were their top 4 trending root causes of outages:

Configuration updates
System Overload
Unexpected Dependencies
Complexity Migrations

Unexpected Dependencies

Let's start with unexpected dependencies and why they are so difficult to identify even for companies like Meta. The problem starts at the tool stack, observability tools are usually the first line of defence however they measure outputs such as metrics, logs & traces. While useful they require a good mental model and a deep understanding of the application in order to interpret them. But what we are talking about is unexpected issues, which often fall outside of our own mental model and/or observability tools.

When these type of outages happen, they can be complex to resolve as the system's behaviour contradicts out understanding of how it should work. This leads to confusion and requires individuals rebuild their mental model of the system on the fly, as mentioned in the brilliant STELLA report. In Meta's case services went down globally for close to six hours.

The solution? Blast Radius.

By measuring config changes (inputs) instead of outputs we can:

Ensure that the configuration and current state of a system are readily accessible.
Enabling users to easily discover the potential impact of their intended changes and what areas might be affected.
Providing users with the means to validate that their modifications have not caused any issues downstream.

Using our GitHub action (or doing it manually) you can go from Terraform Plan → Blast Radius. The blast radius is based on your live AWS state, not Terraform, which lets you see what might break:

Includes resources not managed by Terraform
Discovers dependencies even if they were created manually
Shows live data, not out-of-date CMDB data
Does all of this with read-only access, no agents, no telemetry, and no input from you. If you had to tell us how your apps are architected, we're hardly going to find unexpected dependencies are we?

This means that before making a change, you would be able to identify any potential unexpected dependencies in the blast radius.

What's next? (2/4) Configuration Updates

We're not stopping with just blast radius. Once you've decided to apply your changes, track them with Overmind. Since we've already worked out all the dependencies, we can tell you if your changes has broken something downstream, even if you didn't know it existed.
Want to try it?

Sign up and start calculating blast radius now! It's free for individuals, if you're interested in a team plan contact us.

Overmind is a SaaS Terraform impact analysis tool. It discovers your AWS infrastructure so that it can calculate the blast radius of an application change, including resources managed outside of Terraform. Helping you to identify the causes of outages by showing you which changes caused which problems. While also helping you to deploy changes faster by giving an impact analysis report before any change is made. From this report you can understand if the change can be confidently made, or held back if it's too risky, preventing outages in the first place.

Note: This is beta software & we'd love any feedback. Either by Discord, or book a meeting.

What’s the difference between Terraform Graph and Overmind?

jameslaneovermind — Wed, 12 Jul 2023 14:26:51 +0000

When making changes understanding the output from your Terraform plan is critical to ensuring that there is no unintentional impact from your changes. Because of this Terraform uses terraform graph that helps users to visualise these changes.

What is Terraform Graph?

‍
The terraform graph command is used to generate a visual representation of either a configuration or execution plan. The output is in the DOT format, which can then be used to generate charts.

The recommended program that can read this format is GraphViz, but many web services are also available to read this format. Some add extra formatting to the outputs so that

These are some of the online services:

http://www.webgraphviz.com/
https://dreampuf.github.io/GraphvizOnline/
https://github.com/mdaines/viz.js → emscripten → http://viz-js.com/
https://sketchviz.com/new ‍ ### Usage

First thing install graphviz (example on Mac OS):

brew install graphviz

Then generate a graph output using:

terraform graph [options]

The -type flag can be used to control the type of graph shown. Terraform creates different graphs for different operations. The default type is "plan" if a configuration is given, and "apply" if a plan file is passed as an argument.

Generating Images

The output of terraform graph is in the DOT format, which can easily be converted to an image by making use of dot provided by GraphViz:

$ terraform graph | dot -Tsvg > graph.svg

Here is an example graph output using graphviz:

As mentioned above, there are other services that can convert the provided dot. With these you typically upload the output to the online service and can modify it within the interface.

Heres an example from dreampuf.github..

What is Overmind?

Overmind is a SaaS Terraform impact analysis tool. It discovers your AWS infrastructure so that it can calculate the blast radius of a change including those resources outside of Terraform.

Usage

‍
To get started with Overmind you need to create a account by signing up on the website. Once signed up you'll need to configure a AWS source. Overmind uses a read-only role to query the AWS api to generate the blast radius.

There are two ways of creating a source:

Using Cloud Formation (Automatic):

Using IAM role (Manual):

Once your source is configured you are ready to create your first change.

Now you’ve added the context of the change you need to select the resources that you will be changing. Currently you can do this manually by selecting one or more resource types. A GitHub action is planned to be released soon that will allow you to parse the plan output automatically into Overmind.

Once you’ve selected your types you can then select the individual items you are going to change. Overmind populates these from your AWS source that you configured earlier.

When you’re done selecting the items the final step is to then calculate the blast radius.

Blast Radius

Blast radius queries your AWS infrastructure understanding the relationships and dependencies between different resources and items. From this it can then calculate the impact of your change (or the blast radius).

Once you’ve got the blast radius you can then have a look and see if your change unintentionally impacts anything. By navigating the calculated graph you can explore the links and dive into the meta data to get some context on its configuration.

Terraform Graph vs Overmind?

‍
Before jumping into a comparison of the two it is worth providing some context to the application we are going to be making changes to. It is Kubernetes cluster that manages some API gateways using AWS services such as EKS, EFS & Route53.

Let’s take a look at what this change would look like as a Terraform Graph output. Due to image dimension restrictions the below is a screenshot of a much larger Terraform Graph output.

Within Overmind, by selecting the resource/s that we will be changing, we get the following blast radius.
‍

Conclusion

In this example the output was a 3.5mb svg file or a image with the dimensions 256925px by 3802px. If you were to convert pixels to cm's it's about as it's about as wide as The Wingspan of a 747 (68 metres). Dealing with such a large file is not easy and as you can see from the above screenshot it is not easily readable.

However, Terraform graph can still be a great tool when working with smaller, more manageable changes or if you simply want something to run on CLI. But when you ultimately need to make some larger, more complex changes it is when Overmind would come in to its own. Allowing you to only see the affected resources.

Both tools are freely available:

Terrafrom Graph - docs here.
Overmind - You can sign up here.

What’s the difference between Rover and Overmind?

jameslaneovermind — Wed, 28 Jun 2023 11:58:31 +0000

When using Terraform the output from terraform plan can be used to help understand the impact your infrastructure changes. Terraform has terraform graph to help visualise these changes however when dealing with large or complex infrastructures this can quickly become difficult to navigate. Because of this several visualisation tools have emerged both open source and enterprise.

What is Rover?

Rover is an example of a open source tool that enables users to visualise their Terraform plan. Rover is a interactive Terraform Plan visualiser that helps users explore their state and configuration. Rover is open source and runs locally on your machine. This means that your Terraform state stays local and isn't sent to a remote server for processing. In addition, Rover uses the plan file to generate the state. So in addition to visualising the current infrastructure state, you're able to view any changes to the resources (creation, modification, or deletion).

Usage‍

Rover does this by:

Generating a plan file and parsing the configuration in the user’s root directory.
Parsing the plan and configuration files to generate three items: the resource overview (RSO), the resource map (map), and the resource graph (graph).
Rover then consumes the RSO, map, and graph to generate an interactive configuration and state visualisation hosted on localhost:9000
The quickest way to get up and running with Rover is by using Docker.

Run the following command in any Terraform workspace to generate a visualisation. This command copies all the files in your current directory to the Rover container and exposes port :9000

docker run --rm -it -p 9000:9000 -v $(pwd)/plan.json:/src/plan.json im2nguyen/rover:latest -planJSONPath=plan.json

Once Rover is running on localhost:9000, navigate to it in a browser to find the visualisation.

From here you can then explore the visualisation of your terraform plan output. The legend on the left hand side shows you the different categorisations of resources and other items. With rover you are able to export the generated graph as .SVG file.

What is Overmind?

Overmind is a SaaS Terraform impact analysis tool. It understands all of the dependencies across your AWS infrastructure which means it can calculate the blast radius of a change, even for those resources outside of Terraform.

Usage

To get started with Overmind you need to create a account by signing up on the website. Once signed up you need to add a AWS source. Overmind uses a read-only role to query the AWS api to generate the blast radius.

There are two ways of creating a source:

Using Cloud Formation (Automatic):
Using IAM role (Manual):

Once your source is configured you are ready to create your first change.

You will need to give your change a:

Name
Description
Ticket link
Owner
CC emails (Optional) Now you’ve added the context of the change you need to select the resources that you will be changing. Currently you can do this manually by selecting one or more resource types. A GitHub action is planned to be released soon that will allow you to parse the plan output automatically into Overmind.

Once you’ve selected your types you can then select the individual items you are going to change. Overmind populates these from your AWS source that you configured earlier.

When you’re done selecting the items the final step is to then calculate the blast radius.

Blast Radius

Rover vs Overmind?‍

Before jumping into a comparison of the two it is worth providing some context to the application we are going to be making changes to. It is Kubernetes cluster that manages some API gateways using AWS services such as EKS, EFS & Route53.

Let’s take a look at what this change would look like within Rover. By taking that same Terraform plan output and running it through Rover we get the below.

Within Overmind, by selecting the resource that we will be changing, we get the following blast radius. We are able to see the items related to this resource that we are changing, meaning that we are informed on the impact of this change.

Conclusion

To compare the two, Rover does a great job at visually showing you the output of your Terraform plan. However it can be quite complex to understand with the various dependencies. As Overmind only shows what you will be impacted it is easier to navigate therefore you’re more likely to identify any issues ahead of time.

Both tools are freely available:

Rover - you can find the github repo here.
Overmind - You can sign up here.

What’s the difference between Terraform Plan and Overmind Blast Radius?

jameslaneovermind — Wed, 31 May 2023 16:19:12 +0000

Blast radius is not another Terraform plan visualisation tool

If you’re familiar with Terraform then there’s a good chance you’ve used the Terraform plan command. It compares your current state to your desired state. Building a ‘plan’ that contains a ‘diff’ between both. The output gives us what resources will be created and destroyed along with any modifications before then executing the apply command.

Within the Terraform CLI you’ll see a plan output looking something like this:

And there’s some great tools out there to help you both format and visualise the output so that it is easier to interpret:

Pluralith, Runalantis & Scenery are just some of the great tools out there you can use.

So what’s the problem?

Terraform plan will tell you about the things it’s going to change:

It’ll even tell you if it’s going to change multiple things:

But it won’t tell you the context of those things within the wider application/infrastructure:

You need to be told which pieces you’re touching, sure, and terraform plan is a brilliant way to do that. But you also need to know where those pieces sit in the Jenga tower that is your infrastructure, and what effect removing them might have. That’s what Overmind’s blast radius does.

Overmind blast radius

Overmind understands all of the dependencies within your AWS infrastructure, so we can calculate the blast radius of a change, even for those resources outside of Terraform. Showing you the consequences of your changes, not just the changes themselves.

Start by opening a Terraform pull request and Overmind will discover the dependencies of the things you’re going to change.

Based on what you're changing, Overmind will calculate blast radius of the affected items. Use the graph to explore relationships and dependencies between these items.

The blast radius contains:

What infrastructure will be affected.
What applications rely on that infrastructure.
What health checks those applications have.

After Merging the PR, you’ll automatically get alerted if your change breaks something, Including a diff of exactly what changed, how it's related, and how to change it back.

We need your help

After a successful early access program where we discovered over 600k AWS resources and mapped 1.7 million dependencies. We are now looking for innovators to join our design partner program to help test impact analysis (only for AWS infrastructure at the moment).

If you're interested in getting access before anyone else and influencing the direction of what we're building register here.

Or if you want to be notified the minute we go live join our waiting list here.

Preventing Outages: Limitations of Even the Best Observability and Monitoring Tools

jameslaneovermind — Thu, 25 May 2023 14:28:02 +0000

It was a Friday afternoon and we had planned to roll out a big change that we’d been working on and testing on all week. We knew this was a bad idea, but we were confident! Firstly the change was related to the way the backend UNIX fleet authenticated user logins so should have been fairly innocuous, and we had done all the testing we possibly could, but there was still some risk.

So we pressed the button and rolled out the change. The results came back all green, we could log into the servers, and all that we needed to do was wait. As we waited for all the results to come back, the phone rang.

“Hey, nobody in the department can save PDFs anymore.”

The whole department is at a standstill because they can't save PDFs. We haven’t touched any laptops though, how could we possibly have broken the ability to save PDFs? We started frantically looking into it and it turns out that they aren’t clicking Print -> Save as PDF as you’d expect, they have an actual printer called “PDF Printer” that they print to instead, which we’ve managed to break somehow.

We then tried the easiest things first:

Ask if anyone knows what it is: nobody does
Check if it exists in the CMDB: it doesn’t
Check the wiki: no mention of it

In the end, it turned out that about 10 years ago somebody put a physical server in a data center. And the job of that server was to pretend to be a printer. Meaning that when somebody prints it, it saves it to a pdf, and then it runs a script that picks up that PDF and moves it to a mount point. It didn’t make sense to me at the time, and it still doesn’t, but that’s what we had.

In the end, we managed to get the “printer” working again, but not before everyone in the affected department had already gone home for the weekend without being able to finish their work for the week.

What does this story tell us about the limitations of observability and monitoring tools?

No matter how reliable your systems are or how thoroughly you monitor them, outages can and will occur. Monitoring tools are only as effective as the data points they can access. They can provide valuable insights into system performance but they may not capture everything needed when making a change or finding a root cause fix. A lack of data can make it difficult to pinpoint the cause of an outage, especially when the issue is complex. Often involving multiple systems that can be outside our mental model. These unknown unknowns can be particularly challenging to diagnose and resolve leading to lengthy downtimes.

The typical (wrong) response: Risk Management Theatre

When an outage occurs, a common response is to implement more risk management processes in an attempt to stop the outage from happening again. However, this increased focus on risk management processes results in a substantial increase in lead time. Puppet’s State of DevOps report found that low-performing companies that engaged heavily in risk management theatre had 440x longer lead-times than high-performing organisations.

Companies with these long lead times make 46x fewer changes, meaning that each change needs to be much larger in order to keep up. Less practice, and larger changes means that they are five times more likely to experience failures. When failures do occur, the consequences are much more severe.

The combination of larger changes, decreased frequency, and limited experience in handling such situations leads to a mean-time-to-recovery almost 100x longer than that of high-performing organisations. And remember that it was large outages that caused this in the first place, so the process feeds back on itself, making the company slower and slower.

Answer = Inputs

Observability tools that measure outputs such as metrics, logs & traces require a good mental model and a deep understanding of the application in order to interpret them. But as we’ve already seen, outages are often caused by unexpected issues outside of our own mental model. When this happens, the system’s behaviour contradicts out understanding of how it should work. This leads to confusion and requires individuals rebuild their mental model of the system on the fly, as mentioned in the brilliant STELLA report (Woods DD. STELLA: Report from the SNAFUcatchers Workshop on Coping With Complexity. Columbus, OH: The Ohio State University, 2017).

To address this challenge, we should shift our focus toward measuring inputs. This enables engineers to create new mental models as needed, whether during the planning stage of a change or in response to an outage. Current tools do not adequately support this type of work. When constructing a mental model, we typically rely on "primal" low-level interactions with the system, often accomplished through the command line, which demands a great deal of expertise and time. To resolve this issue, we must find a way to expedite the process of building mental models by measuring input or configuration changes instead.

If we are to solve this, we must make building mental models much faster, meaning:

Ensuring that the configuration and current state of a system are readily accessible.
Enabling users to easily discover the potential impact of their intended changes and what areas might be affected.
Providing users with the means to validate that their modifications have not caused any issues downstream.

By measuring config changes (inputs) instead we can understand context on demand and have the confidence that our changes won’t have any unintended negative impacts.

Impact analysis

At Overmind, we’ve been building a solution that addresses these challenges. By making the system's configuration and state easily accessible empowering users to confidently make changes without the fear of things going wrong.

Open a pull request

Start by opening a Terraform pull request and Overmind will discover the dependencies of the things you’re going to change. No lengthy scanning processes or agents involved in the setup.

Calculate blast radius

Based on what you're changing, Overmind will calculate blast radius of the affected items. Use the graph to explore relationships and dependencies between these items.

The blast radius contains:

What infrastructure will be affected.
What applications rely on that infrastructure.
What health checks those applications have.

View diffs & validate health

Automatically get alerted if your change breaks something. Including a diff of exactly what changed, how it's related, and how to change it back. Spend less time on validation and letting changes “bake” meaning a faster time to production.

Quickly identify which changes caused a problem.
Compare the difference between configurations to uncover the root cause.
Minimise downtime caused by application breaking changes.

Design Partner Program

After a successful early access program where we discovered over 600,000 AWS resources and mapped 1.7 million dependencies. We are now looking for innovators to join our design partner program to help test impact analysis.

Start by sharing your impact analysis goals with the Overmind team, and see if our program is the perfect fit for you.

If you're interested in getting access before anyone else and influencing the direction of what we're building register here.

Or find out more about Overmind and join our general waiting list here.

8 Effective Tips to Reduce Your EC2 Spending

jameslaneovermind — Mon, 22 May 2023 15:40:16 +0000

Introduction

If you're running a business that relies heavily on Amazon EC2, you're probably well aware of the costs that come along with it. Cloud infrastructure can be expensive, and EC2 is no exception. However, there are a number of proven best practices that you can implement to help reduce your EC2 spending. In this article, we'll explore those practices in detail.

1. Identify Amazon EC2 instances with low utilisation and reduce cost by stopping or rightsizing‍

One of the most impactful ways to reduce AWS EC2 costs is to right-size your instances.

Use AWS Cost Resource Optimization to get a report of EC2 instances that are either idle or have low utilisation.
Reduce costs by either stopping or downsizing these instances.
Use AWS Instance Scheduler to automatically stop instances.
Use AWS Operations Conductor to automatically resize the EC2 instances (based on the recommendations report from Cost Explorer).
Use AWS Compute Optimizer to look at instance type recommendations beyond downsizing within an instance family. ‍ ### 2. Use Compute Savings Plans to reduce EC2, Fargate and Lambda costs‍

Compute Savings Plans automatically apply to EC2 instance usage regardless of instance family, size, AZ, region, OS or tenancy, and also apply to Fargate and Lambda usage.

Use one year, no upfront Compute Savings Plans to get a discount of up to 54% compared to On-Demand pricing.
Use the recommendations provided in AWS Cost Explorer, and ensure that you have chosen compute, one year, no upfront options.
Once you sign up for Savings Plans, your compute usage is automatically charged at the discounted Savings Plans prices. ‍

3. Use Amazon EC2 Spot Instances for AWS Cost Reduction‍

If your workload is fault-tolerant, using Spot Instances can reduce EC2 costs by up to 90% and is a key strategy for AWS cost reduction.

The main caveat is that spot instances can be interrupted at two minutes’ notice.

Examples of typical workloads include big data, containerised workloads, CI/CD, web servers, and development/testing.

Amazon provides the Spot Fleet feature, which lets you run both on-demand and spot instances in the same Auto Scaling group, letting you reserve some on-demand instances which cannot be interrupted, for critical components.
‍

4. Analyse and attribute expenditure‍

Clouds can help you accurately identify the cost and usage of systems, allowing transparent attribution of IT costs to the individual workload owners.

It enables you to measure return on investment (ROI) and helps workload owners optimise resources and reduce costs.
‍

5. Use the latest instance types‍

Switching your EC2 Instance to the latest Instance types can reduce AWS cost up to 6%. This is because they typically provide higher efficiency or better performance at a lower price.
‍

6. Purchase reserved instances for a reasonable period of time‍

Simply put, if you don't need an instance, don't keep it longer than necessary.
‍

7. Schedule instances to ensure they run only during business hours or when needed‍

You can achieve this automatically using AWS Instance Scheduler or other tools.

8. Adopt a consumption model‍

AWS recommends paying only for the required computing resources and increasing or decreasing usage according to business needs.

For example, staff typically use development and testing environments eight hours per day during a workweek. You can potentially achieve 75% cost savings by stopping these resources when they are not used.

Conclusion

Reducing AWS EC2 costs is an essential aspect of optimising your cloud infrastructure and maximising your ROI. By implementing these 8 best practices, you can ensure that you’re only spending what you need too.

6 Ways to Verify IP Addresses Across Your AWS Accounts

jameslaneovermind — Thu, 18 May 2023 12:47:34 +0000

Introduction

Verifying IP addresses used across multiple AWS accounts is important for maintaining IP reputation and customer whitelisting. To collect information about networking resources in a multi-account AWS environment, there are several scalable options including IPAM, AWS Config, and building an automated solution using the AWS CLI or SDKs.

1. Instance Metadata API‍

‍Use the Instance Metadata API on the instance to retrieve the AWS account ID associated with the instance. The API can be accessed by running the following command on the instance:

curl http://your-ip-address/latest/dynamic/instance-identity/document

The command will return a JSON object that contains the account ID in the accountId field.

2. AWS CLI and Network Interfaces‍

Use the AWS CLI to find the network interface that owns the IP address. The following command can be run:

aws ec2 describe-network-interfaces --filters Name=addresses.private-ip-address,Values=

Replace <IPv4 address> with the IP address that needs to be investigated. The output of the command will include the description of the network interface, which can be used to identify the resource that owns the IP address.

3. Amazon VPC IP Address Manager (IPAM)

Use Amazon VPC IP Address Manager (IPAM) to monitor and audit IP addresses at scale. IPAM is a VPC feature that makes it easier to plan, track, and monitor IP addresses for AWS workloads. From a centralised dashboard, IPAM can monitor IP address space that's in use and use the IP historical data to search for the status change of IP addresses or CIDRs. IPAM can be integrated with AWS organisations to activate the Amazon VPC IPAM service to manage and monitor networking resources created by all AWS organisations member accounts.

4. Automating with AWS Config Aggregator

Automate multi-account network information capture with AWS Config aggregator. AWS Config records AWS resource configurations and allows reviewing changes in configurations and relationships between AWS resources. Networking resource types collected by AWS Config include internet gateways, NAT gateways, load balancers, VPC CIDRs, and subnets. AWS Config data from multiple accounts and AWS Regions can be aggregated into a single account using multi-account, multi-Region data aggregation.

5. Automatic Report Generation with aws-sts-network-query-tool‍

Automatically generate a multi-account network resources report using aws-sts-network-query-tool. This script uses AWS CLI to collect networking-related information for multiple accounts and outputs the information in a CSV. It can be run by specifying a list of member accounts to scan. The script can collect information about internet gateways, NAT gateways, load balancers, VPC CIDRs, subnets, elastic IP addresses, and elastic network interfaces.

6. Overmind‍

Overmind makes it easy to search IP addresses across accounts while getting all important related content and metadata about its usage. Within Explore, you can start by listing all ec2-address types. From there you can expand outwards discovering any related types including DNS entries, ec2-network-interfaces or IPs.

To do this across multiple accounts simply those accounts as new sources. Once you have added these sources you will be able to search across all of them by making sure the scope is set to ‘*’.

Choosing the Right Option‍

Choosing the option depends on the specific use case and the available infrastructure.

For a list of VPC CIDRs, subnets, elastic network interfaces, elastic IP addresses, and IP usage within CIDRs, IPAM can be used.

If internet gateways, NAT gateways, and load balancers within accounts need to be listed, along with the resources elastic IP addresses are attached to, AWS Config aggregator or aws-sts-network-query-tool can be used.

If the accounts are not within an AWS organisation, aws-sts-network-query-tool would be the best option.
‍

FAQs

What is the Instance Metadata API, and how does it help in verifying IP address usage?

To verify IP address usage, you can utilise the instance metadata to get the instance ID and other relevant details. The instance metadata is available in all Amazon EC2 instances at the following address: /latest/meta-data/. In there, you can find quite a lot of useful information, including the instance ID, which can be used to verify IP address usage.

Can I use the AWS CLI to find the owner of an IP address in AWS?

Yes, you can use the AWS CLI to find the owner of an IP address in AWS. The docs link can be found here.

Which option should I choose if I need to monitor IP addresses within VPCs and track their usage?

To monitor IP addresses within VPCs and track their usage, the best option is to use Amazon VPC IP Address Manager (IPAM).

Calling out for all those who work with AWS

jameslaneovermind — Tue, 16 May 2023 14:10:31 +0000

What is Overmind? (In a nutshell)

Make AWS changes with confidence. Start by submitting a pull request → discover the blast radius → track and validate that your change has not broken anything. Using read-only AWS access and no other inputs from you.

What are we looking for?

After a successful early access program where we discovered over 600k AWS resources and mapped 1.7 million dependencies. We are now looking for innovators to join our design partner program.

Whether you're making just a couple of tweaks or a few hundred changes in AWS, our design partner program is for those who are seeking innovation and the opportunity to shape the future of application changes.

By becoming part of the program, you'll join a community of brilliant minds, all driven by the same passion for tech. Together, you’ll get to try the latest version before anyone else.

How do I get started?

Start by sharing your impact analysis goals with the Overmind team, and see if our program is the perfect fit for you.

If you're interested in getting access before anyone else and influencing the direction of what we're building register here: https://app.reclaim.ai/m/dylan-overmind/design-partner-chat

Or find out more about Overmind and join our waiting list here: https://overmind.tech

How to choose the right EC2 Instance for your application

jameslaneovermind — Thu, 11 May 2023 14:02:01 +0000

TLDR: With over 400 instances to choose from its not easy. These 6 things will help make narrowing down your decision that bit easier.

As a DevOps engineer working with AWS, selecting the right Elastic Compute Cloud (EC2) instance type can be a daunting task given the almost 400 options available. Nevertheless, there are several things you can do to simplify the process and optimise costs while delivering high performance. Making careful choices when it comes to EC2 instances is crucial, as compute often constitutes the bulk of cloud bills. By optimising this aspect, you could potentially achieve significant reductions in your cloud costs.

AWS has a great video on their youtube filmed at reInvent:2021 that talks you through in detail selecting and optimising instances. However I've summarised some of the top 6 below. Adopting this will help you when it comes to selecting your next instance.
‍

The top 6 things to consider when making a decision

1. Identify your application requirements

Determine the workload's minimum requirements to avoid over-provisioning or running memory-intensive applications on underpowered instances. Choose an EC2 instance that satisfies the application's needs, including vCPU count, vCPU architecture, network, memory, and SSD storage.

For example, if you intend to support a machine learning application, choose a GPU-dense instance type instead of a CPU-dense one. In 2021 AWS released EC2 Inf, an instance type designed for inference, which provides higher throughput and a lower cost per inference than EC2 G4 instances.
‍

2. Research EC2 instance types and families

Many EC2 instance families exist, each with its strengths and weaknesses. Conduct a thorough search for the most appropriate family for your workload needs. For example, the C5 family might be the best choice for a high-performance computing (HPC) workload. If you require high network performance, consider the enhanced networking feature on the C5n family.

Remember to have cost savings in mind when selecting an instance size. EC2 instances are available in various sizes, and you should scale your resources to match the workload requirements. However, remember that size is not the only factor affecting cost. AWS deploys different computers to provide compute capacity, each with chips having varying performance characteristics. Benchmarking is a reliable method of determining the most cost-effective instance size.
‍

3. Consider the pros and cons of different pricing mode

AWS offers several pricing models for EC2 instances, including Spot Instances, Reserved Instances, and On-Demand Instances. Each has its advantages and disadvantages, and you should select the one that best meets your workload needs and budget.

On-Demand Instances:
Customers pay for actual usage on an hourly basis, with no long-term commitments or upfront payments required. Advantages include flexibility, scalability, and the ability to pay only for what you use. However, it can be more expensive for long-running instances.

Reserved Instances:
Customers receive a significant discount on the hourly rate in exchange for committing to using a specific instance type for one or three years. This pricing model is advantageous for stable workloads that run continuously and offer cost savings over the long term. However, it can be inflexible.

Spot Instances:
Customers can bid on unused EC2 capacity, with prices fluctuating based on supply and demand. This pricing model offers potential cost savings, but the availability of Spot Instances can be unpredictable. AWS can also terminate instances with two minutes' notice when the spot price exceeds the customer's bid.
‍

4. Be mindful with your storage selection

When selecting an EC2 instance in AWS, consider factors such as performance, durability, cost, storage type, and storage size. AWS has two types of storage: block and object. Block storage is for consistent and low-latency performance. AWS has Elastic Block Store (EBS) for persistent block storage, and instance store for temporary block storage. Object storage is for scalable and durable storage. AWS has Simple Storage Service (S3) for highly durable and scalable storage, and Amazon Glacier for low-cost storage.
‍

5. Use Spot Instances (even for production workloads)

Spot Instances are unused EC2 instances available at a lower price than on-Demand instances. Although they may be interrupted if the demand for EC2 instances rises, they can be an excellent option for production workloads with flexible timing.
‍
Explained: What are Spot instances?
Spot Instances are a cost-effective way to run your applications because you can bid on unused instances. The price is lower than the on-demand price and varies based on supply and demand.
‍

6. Automate everything

To improve your system's efficiency and consistency, simplify the selection and deployment of EC2 instances. Use automation tools like Amazon Elastic Container Service (ECS) and Lambda. These tools automate the entire process from instance selection to ongoing monitoring. By automating, you can save time and effort while ensuring peak performance and avoiding human error.

In summary, selecting the optimal EC2 instance type may seem intimidating given the vast number of options available. Still, by following the above steps and taking into account factors such as cost optimisation, storage selection, and pricing models, you can make an informed decision.

The Hidden Complexity in Your Cloud Architecture Diagrams

jameslaneovermind — Wed, 10 May 2023 15:07:51 +0000

As companies migrate their applications to the cloud, they can find their architecture diagrams become increasingly complex. These diagrams provide a visual representation of the various components and how they interact with each other. However, they may not accurately reflect the true complexity of the system.

During early access, our team had the opportunity to analyse 2.3 million AWS resources and dependencies to quantify the complexity hidden in architecture diagrams. What we found on average was 3 links for every resource. A ratio not often found in even some of the most complex architecture diagrams. Let’s take a look at why that’s a problem.

The problem *with complexity (explained by looking at houses)*

A house plan like the one above is great for giving you a visual representation of the layout and features (number rooms, amenities etc.) However, say you wanted to add a extension or even drill a hole in the wall. Would you feel confident that you everything required to not knock down a structural wall or drill through a gas line?

Instead you might consider consulting the building plans or blueprint. They contain the information you need to confidently make your decisions. However while very useful, blueprints can be complex containing lots of measurements and annotations and if you don’t know what you’re looking for may cause more issues.

The same can be said for architecture diagrams…

AWS diagrams like the one above are a great tool for onboarding new engineers or communicating a high-level overview to stakeholders. They give a clear but often concise representation of an application that does not require much prior experience or context to understand.

But would you feel confident making a change to your application based on the above? Knowing from what we’ve already said above about hidden complexity.

Even changing something simple like a security group could be problematic. The architecture diagram may show you some connections. But there could be other EC2 instances or RDS databases that are also using that security group. If you make change, it could impact those resources.

More is not always the answer

Does that mean the answer is to generate a digram mapping out every link and resource that are related to the application that we are making changes to? To show you what that would look like on the same EKS cluster we can run a query in Overmind’s explore feature. We can set the link depth so that will discover all the relationships & links to other resources.

What you can see is that same application actually has:

164 related items
39 related resource types

Which is much more that what our diagram was telling us. Meaning that now if we wanted to make a change we can see everything that could be impacted. The resources, items links and meta-data in one diagram.

But when you’re dealing with this level of detail it becomes a challenge to display and navigate easily in a interactive GUI let alone trying to replicate the same by drawing a static architecture diagram.

Which leaves us in a difficult position because in order to confidently make changes we need to know what will be impacted. And to know that we need to map out all links to the resource we are changing. But from what we’ve seen when even a simple application has that many related resources and links it can become a challenge to work with.

The solution

It is precisely this challenge that has led us build Overmind. With impact analysis you don’t need to worry about creating a diagram of your entire application architecture. Tell it what you’re going to change and you will be informed of any resources outside of that scope that have been impacted by that change. Meaning that you will have the confidence that changes you make won’t have any unintended consequences.

We are currently looking for design partners to join our waiting list. Sign up here → https://overmind.tech

Confidently working with IAM Roles in AWS

jameslaneovermind — Fri, 28 Apr 2023 09:39:14 +0000

IAM roles

With more than 400 million operations per second AWS IAM usage is on a scale that is often hard to comprehend. Combine that with AWS still maintaining its majority share of the cloud market, it's fair to say a good chunk of the internet is regulated by IAM roles and policies.

With that being the case you can be confident that AWS has a depth of expertise and wisdom backing up IAM. But it doesn't make managing IAM roles less of a arduous task. Often it means spending your time scrolling through multiple lines of JSON entries, which isn't exactly the most efficient way to look at permissions.

The problem is only made worse in large organisations with several accounts and multiple services. It can be uphill battle to keep track of all the permissions assigned to different IAM roles. Making assigning the correct permissions a challenge if you don't know your resources and services like the back of your hand. While also making retrospective tasks such as auditing time consuming as you struggle to get the context you need to make important decisions.

Working with IAM roles

Auditing roles

There’s a number of different reasons why you’d need to audit IAM roles. As part of a new project to ensure that no unused roles haven’t been created and forgotten. As part of compliance, ensuring you meet regulatory requirements. Or even as part of a security or cost review. Ensuring that unused roles are cleaned up and users have the appropriate level of access is vital and can also help ensure users are held responsible for their actions.

In AWS

To do this you can check the last time each role made a request to AWS and use this information to determine whether the team is using the role. You want to gather more information about the role’s access patterns to determine whether you ought to delete it.

From the role detail page, navigate to the Access Advisor tab and investigate the list of accessed services and verify what the role was used for.

In the access Advisor tab you can investigate the list of accessed services and verify what the role was used for.

This can also be done via the CLI:

$ aws iam generate-service-last-accessed-details --arn arn:aws:iam::1234567:role/role-name
{
    "JobId": "10c3dc31-6ccc-69d2-1185-91e9ad363831"
}

$ aws iam get-service-last-accessed-details --job-id 10c3dc31-6ccc-69d2-1185-91e9ad363831
{
    "JobStatus": "COMPLETED",
    "JobType": "SERVICE_LEVEL",
    "JobCreationDate": "2023-04-25T12:28:18.712000+00:00",
    "ServicesLastAccessed": [
        {
            "ServiceName": "AWS Security Token Service",
            "LastAuthenticated": "2023-04-25T11:49:09+00:00",
            "ServiceNamespace": "sts",
            "LastAuthenticatedEntity": "arn:aws:iam::944651592624:role/aws-source-pod",
            "LastAuthenticatedRegion": "eu-west-2",
            "TotalAuthenticatedEntities": 1
        }
    ],
    "JobCompletionDate": "2023-04-25T12:28:20.485000+00:00",
    "IsTruncated": false
}

The question often remains is this information enough to make important decisions on. For example:

Can I remove this IAM role that has not been used in 89 days? What happens if it is part of a service that is used every 120 days?
How can I be sure that I can I remove a role that has no activity?
I know the role was used, but which AWS resource used it? A lambda function? An EKS pod?

Context is key.. but often missing

What’s missing in both the above questions is context. Context of the role and if it is linked to anything. The problem with context is that it is often difficult to get without years of experience or up-to-date CMDB’s/ documentation.

A solution with Overmind

Using Overmind does not require any of the above. In fact, it was built to be used with no prior context. You can simply search for what you want, in this case ‘iam-role’. Overmind will do the work finding them even across multiple regions and accounts.

From here we can quickly distinguish any unused roles or policies because they won’t be linked to any other resources.
For example a unused role will look like this.

Whereas one that is currently in use will have links to other resources and look like this.

In Overmind you can expand out and discover what other resources it is linked to or being used by. Providing us with the context we were missing before.

From here we’ll be able to understand what this application is and the resources it needs to work out. Answering the question of what the impact would be if we were to remove these roles or polices. Now we have the missing context we can go ahead and proceed confidently knowing that our changes won’t have any unintended impact.

We aren’t stopping here..

What if before making a application change Overmind could work out the blast radius and inform you on the impact? Or after a unsuccessful change you could go back and see a snapshot of what your application looked like before? Across all your AWS resources.

That’s what we are building and we would love you to join us on this journey. Add your name to our waiting list to register your interest → https://overmind.tech