How to search your AWS accounts for any resource

Retrieving information about resources you have deployed in your Amazon Web Services (AWS) infrastructure means tediously navigating the AWS Management Console or using the AWS Command Line Interface.

This approach works well in a single account setup. Yet the best practice proposed by AWS is to set up a multi-account environment to separate your workloads and users. As the number of accounts grows, navigating your infrastructure and finding resources via the Console or the CLI becomes increasingly difficult.

The number of resources in these accounts just keeps growing. Developers create resources using tools such as Terraform, CDK, or CloudFormation… or sometimes even the console or CLI.

It's not just the resources themselves, it's also the relationships between your resources that are relevant: an EBS volume is mounted to an EC2 instance running in a VPC and reachable via an ALB load balancer, for example.

So how can you see everything that is running in your cloud, including the dependencies between your resources?

Graph-based Search

We created Resoto to allow the user to effortlessly search resources and automate workflows. Resoto gathers data about your infrastructure and builds a directed acyclic graph, where resources are vertices and their relationships/dependencies edges.

This graph is what makes Resoto so powerful. But we also needed a way to allow users to query this data.

Graph data is not relational, so SQL was not a good fit. And existing graph query languages like Cypher, Gremlin, or GSQL have steep learning curves and are unnecessarily complex for this use case.

And so we developed our own search syntax tailored specifically to Resoto. The Resoto Shell allows you to interact with your Resoto installation. In particular, it provides a search command.

Example: Search for EC2 instances

Let's try searching for all available EC2 instances. is() will match a specific or abstract type in a polymorphic fashion, checking all types and subtypes of the provided type.

The instance_cores filter will limit results to only those instances with more than two cores. The query below will automagically search your entire infrastructure, regardless of account or region!

search is(aws_ec2_instance) and instance_cores > 2

and here the (abbreviated) result:

id=i-a..., name=crmsec, age=2y2M, account=dev, region=us-east-1
​id=i-0..., name=airgap, age=2M, account=staging, region=eu-central-1
​id=i-0..., name=flixer, age=1M3w, account=sales, region=us-west-2

The query found three instances in three accounts and three regions. The default output is a condensed list view, but it is also possible to get all collected properties of any resource using the dump command:

search is(aws_ec2_instance) and instance_cores > 2 limit 1 | dump

In the case of EC2, these properties for example are the number of cores, memory and the actual instance type and its status:

reported:
​  kind: aws_ec2_instance
​  id: i-a...
​  tags:
​    aws:cloudformation:stack-name: lk-build-server
​    aws:cloudformation:stack-id: arn:aws:cloudformation:...
​    owner: team-proto
​  name: LKbuild
​  instance_cores: 4
​  instance_memory: 16
​  instance_type: t3.xlarge
​  instance_status: stopped
​  age: 1y10M

We can refine and group the results. Let's group our instances by instance_type using the count command:

search is(aws_ec2_instance) and instance_cores > 2 | count instance_type

t3.2xlarge: 1
​m5.4xlarge: 15
​total matched: 16
​total unmatched: 0

The search returns sixteen EC2 instances, including fifteen m5 and one t3 xlarge.

Using graph search to understand dependencies

Now, let's say we want to find all ELB load balancers attached to the EC2 instances returned above. We first need to understand Resoto's graph data structure to tackle this problem.

When Resoto collects data on your cloud infrastructure, it creates an edge between ELB and EC2 instances if the ELB balances the traffic of the related EC2 instance. In the image below, you can see the how the graph captures the entire set of dependencies in the account:

search is(aws_ec2_instance) and instance_cores > 2 --> is(aws_elb)

name=a5..., age=1y1M, account=sales, region=eu-central-1
​name=a3..., age=6M2w, account=staging, region=us-west-2

The -->arrow will take all matching EC2 instances and walk the graph "outbound," moving precisely one step. The list of matching items is not limited only to ELB load balancers, so with is(aws_elb) we filter the list again to return only ELB results.

It's also possible to reverse the last query to output all EC2 instances behind an ELB:

search is(aws_elb) <-- is(aws_ec2_instance) and instance_cores > 2

id=i-0..., name=airgap, age=2M, account=staging, region=eu-central-1
​id=i-0..., name=flixer, age=1M3w, account=sales, region=us-west-2

The arrow is now mirrored and traverses the graph "inbound," walking edges in the opposite direction.

Use cases

Graph-based search becomes useful when you're trying to solve for problems that require understanding of how resources are connected to each other.

An example is the "blast radius" of a resource.

When you're looking at cleaning up an EC2 instance, what other resources are you taking down as well? Or, the other way around, what resources do you need to clean up first before you want to clean up an unused EC2 instance?

Try it yourself!

These examples only scratch the surface of Resoto's search syntax. While this post is AWS specific, we also support GCP and DigitalOcean.

Resoto is open source, self-hosted and free to use! Check out our documentation and give Resoto a spin!

---

This post was originally published by my colleague Matthias Veit at https://resoto.com/blog/2022/02/04/resoto-search-101 on February 4, 2022.