DEV Community: Chris Farris

Streamlining incident response investigations with Steampipe relationship graphs

Chris Farris — Mon, 23 Jan 2023 14:23:07 +0000

We covered how Steampipe can assist with Detection and Analysis in Enrich Splunk events with Steampipe. Today we’re going to look at how leveraging Steampipe relationship graphs can assist in the Containment, Eradication, and Recovery phases of the incident response cycle.

NIST 800-61 Incident Response Cycle

Let's say you have evidence a cloud host was compromised. One of the key questions you need to answer is, “What could that attacker have done while inside our network?”

Steampipe’s new relationship graphs are one method to determine the compromised host’s connectedness and Lateral Movement possibilities.

Start with EC2 Instance Detail

A typical incident starts with an event, and that event is often associated with an EC2 Instance. We can select the suspected compromised host via the AWS EC2 Instance Detail. In addition to details about the instance size, IP address, and tags, you can see several related resources including the SSH KeyPair used to launch the instance, the EBS Volume, and most importantly the IAM Instance Profile and VPC Information.

Here we see that the compromised instance has the RenderRole-dev IAM role and is located in the VPC-Public2-Subnet in the Dev VPC.

Dive into the Render Role

Click RenderRole to visit the AWS IAM Role Detail dashboard.

The IAM Detail relationship diagram shows us the attached policies, along with the EC2 instances and Lambda function that use the role. These are the permissions that the attacker could get from the EC2 Metadata Service.

Dive into the Dev VPC

In addition to viewing IAM actions an attacker might have had access to, you probably also want to know what network resources the compromised host could connect to. From the original EC2 Detail, we can also click into the AWS VPC Detail for the Dev VPC.

The relationship diagram shows us that in this VPC was an RDS database, a mail relay server, and a payments server. Critically, there were also VPC peering connections to two other VPCs: the Security VPC and the Meme Factory VPC. Let’s click on the Security VPC to see what is in there.

Dive into the Security VPC

Because Steampipe was configured to scan the entire AWS organization, we can pivot our view across multiple AWS accounts without logging in to the AWS console a second time. When investigating an incident, a responder often has multiple browser windows open to investigate, and this can lead to mistakes. Relationship graphs are an easier and more reliable way to explore the entire cloud network.

In the Security VPC, we find the SIFT forensics server, the Fooli Organization Steampipe instance, and Splunk, which is used as the Fooli SEIM.

Preparation

For incident response, preparation is vital. In order to be ready to use Steampipe to investigate an incident, you need to ensure everything is set up properly.

Follow the guidance on Using Steampipe CLI with AWS Organizations
Download the AWS Insights Mod for Steampipe
Run the Steampipe CLI with steampipe dashboard
If using Steampipe Cloud, simply install the AWS Insights mod in your workspace to get started.

Let's go exploring!

Once you’ve set up Steampipe or Steampipe Cloud to investigate resources, there are many other exciting things you can identify visually that you might miss in a tabular report.

The Steampipe ecosystem now offers more than 90 plugins; each provides several, dozens, or even hundreds of tables. You've always been able to use SQL to join those tables, not only within an API family like AWS, but across diverse APIs. Now you can also use SQL to build relationship graphs across APIs and cloud tenants. Imagine an always accurate network map that links your AWS and Azure VPCs with the firewalls in your data center. The sky's the limit, and we look forward to hearing about your discoveries in our Slack community.

Leveraging CodePipeline to deploy Terraform

Chris Farris — Sun, 22 Jan 2023 18:25:49 +0000

I've been on the CloudFormation side of the IaC Wars since 2014, when I started working in AWS. I've dabbled in terraform but never made it my primary IaC choice. For the Fooli Meme Factory, I needed to mess things up and then quickly revert the state to what it was at deployment time. So for SECCDC 2023, I ported Meme Factory almost entirely over to Terraform.

This led me to two problems. The first was the perennial issue I've had with Terraform from day one: "How do I manage state?". The second issue was how do I leverage some form of CI/CD tooling to allow me to leverage one of Terraform's biggest strengths - the terraform plan capability. Since Fooli is an AWS product, I figured that I should be able to use AWS native tools for this. I've used CodePipeline in the past to preview change-sets with aws-org-formation, so I thought it would be easy to find a well-worn pattern from AWS on doing it.

Apparently, there is no canonical way to use Terraform in CodeBuild, with CodePipeline as the method to review plans before applying them!!! Seriously, WTF?

This made me sad. So I decided to do it my damn self. And now I'm documenting it here for everyone else.

This solution will provide the following:

CloudFormation Template (CFT) to deploy a CodePipeline, CodeBuild Projects, and an S3 Bucket for state and artifact handling.
Buildspec files to perform the terraform plan and terraform apply steps.
Makefiles because I'm old school like that.

Why a CloudFormation template for step 1? To get around the chicken-and-egg problem with state. The CFT will deploy and do the needful to get the account setup for the terraform pipeline without needing a terraform pipeline already in place

How it works.

When a push is made to a monitored GitHub repo, the CodePipeline will trigger. AWS's CodeStar Connections are used to manage the integration between GitHub and CodePipeline. (As an aside: CodeStar connections are so under-the-radar I can't even find a product page to link to. Just some API docs and the above blog post.) Anyway - CodeStar is a much better solution than previous methods that required overly-privileged GitHub Personal Access Tokens to be uploaded into shared AWS Accounts.

So CodeStar Connections will monitor GitHub and fire the pipeline on a push to the specified branch. After that, the pipeline will execute the following stages:

Source Stage - where it downloads the code package from GitHub and stores it in the S3 Bucket.
Terraform Plan Stage - where CodeBuild will execute the terraform plan and copy the tfplan into S3
Review Stage - sends a message to SNS, which (if configured) will email someone to review the output of the Terraform plan in CodeBuild.
Apply Stage - If approved, this stage will fire up CodeBuild to do the terraform apply on the preexisting tfplan file.

CodePipeline & CodeBuild

The CodePipeline is defined entirely in the CloudFormation Template. The CodeBuild projects are created by the CloudFormation Template, but the commands to be executed are defined in the build spec files.

Terraform Plan

The definition of the Plan stage in CodePipeline is:

- Name: terraform-plan
  Actions:
    - Name: terraform_plan
      RunOrder: 1
      Namespace: TfPlan
      InputArtifacts:
        - Name: GitHubCode
      OutputArtifacts:
        - Name: TerraformPlan
      ActionTypeId:
        Category: Build
        Provider: CodeBuild
        Owner: AWS
        Version: '1'
      Configuration:
        ProjectName: !Ref TerraformPlanProject
        EnvironmentVariables: !Sub |
          [
            {"name": "EXECUTION_ID",    "value": "#{codepipeline.PipelineExecutionId}"},
            {"name": "BRANCH",          "value": "#{GitHubSource.BranchName}"},
            {"name": "REPO",            "value": "#{GitHubSource.FullRepositoryName}"},
            {"name": "COMMIT_ID",       "value": "#{GitHubSource.CommitId}"},
            {"name": "env",             "value": "${pEnvironment}"}
          ]

And the Project is:

TerraformPlanProject:
  Type: AWS::CodeBuild::Project
  Properties:
    Name: !Sub ${AWS::StackName}-tf-plan
    Artifacts:
      Type: CODEPIPELINE
    Source:
      Type: CODEPIPELINE
      BuildSpec: buildspec-tf-plan.yaml
    Environment:
      ComputeType: BUILD_GENERAL1_SMALL
      Type: LINUX_CONTAINER
      Image: !Ref BuildImageName
    ServiceRole: !GetAtt ProjectServiceRole.Arn

The EnvironmentVariables in the pipeline are passed into the CodeBuild Project. CodePipeline substitutes environment variables that begin with a # at execution, and the ones beginning with $ are substituted by CloudFormation at deployment. The BuildSpec exports some environment variables, and they're stored in the pipeline under the TfPlan Namespace (Line 5). The BuildSpec part of the CodeBuild Project (Line 9) defines the commands that CodeBuild will execute. That file looks like:

version: 0.2

env:
  exported-variables:
    - BuildID
    - BuildTag

phases:
  install:
    commands:
      - "curl -s https://releases.hashicorp.com/terraform/1.3.6/terraform_1.3.6_linux_amd64.zip -o terraform.zip"
      - "unzip terraform.zip -d /usr/local/bin"
      - "chmod 755 /usr/local/bin/terraform"
  pre_build:
    commands:
      - "make tf-init"
  build:
    commands:
      - "make tf-plan"
      - "export BuildID=`echo $CODEBUILD_BUILD_ID | cut -d: -f1`"
      - "export BuildTag=`echo $CODEBUILD_BUILD_ID | cut -d: -f2`"

artifacts:
  name: TerraformPlan
  files:
    - terraform/$env-terraform.tfplan

Lines 4-6 indicate that we will export two environment variables we want to pass back to CodePipeline, BuildID and BuildTag. These are needed to build the URL for reviewing the plan. The artifacts section on line 23 defines the files created that CodeBuild/CodePipeline should store in S3 and pass between the pipeline stages.

Review Stage

The review stage consists of a manual step and looks like this:

- Name: Review-Plan
  Actions:
    - Name: review-plan
      RunOrder: 1
      ActionTypeId:
        Category: Approval
        Provider: Manual
        Owner: AWS
        Version: '1'
      Configuration:
        NotificationArn: !Ref PipelineNotificationsTopic
        CustomData: "Review the Terraform Plan"
        ExternalEntityLink: !Sub "https://${AWS::Region}.console.aws.amazon.com/codesuite/codebuild/${AWS::AccountId}/projects/#{TfPlan.BuildID}/build/#{TfPlan.BuildID}%3A#{TfPlan.BuildTag}/?region=${AWS::Region}"

Here we construct the ExternalEntityLink from the BuildID and BuildTag from the plan stage. Again variables that begin with a # are substituted by CodePipeline at execution and the ones beginning with $ are substituted by CloudFormation at deployment. We send a message to the PipelineNotificationsTopic which triggers an email to the user:

Apply Stage.

The apply stage is similar to the plan. In CodePipeline it looks like this:

- Name: ExecuteTerraform
  Actions:
    - Name: terraform-apply
      RunOrder: 1
      InputArtifacts:
        - Name: GitHubCode
        - Name: TerraformPlan
      ActionTypeId:
        Category: Build
        Provider: CodeBuild
        Owner: AWS
        Version: '1'
      Configuration:
        ProjectName: !Ref ExecuteTerraformProject
        PrimarySource: GitHubCode
        EnvironmentVariables: !Sub |
          [
            {"name": "EXECUTION_ID",    "value": "#{codepipeline.PipelineExecutionId}"},
            {"name": "BRANCH",          "value": "#{GitHubSource.BranchName}"},
            {"name": "REPO",            "value": "#{GitHubSource.FullRepositoryName}"},
            {"name": "COMMIT_ID",       "value": "#{GitHubSource.CommitId}"},
            {"name": "env",             "value": "${pEnvironment}"}
          ]

In this case, we're inputting the input artifacts from both GitHub and the plan (lines 5-7). We set the GitHubCode as the PrimarySource on line 15, and that becomes the working directory. The other files are written to a different directory, and we have to move them in the BuildSpec file (line 9 below).

The BuildSpec for the apply looks like this:

version: 0.2

phases:
  install:
    commands:
      - "curl -s https://releases.hashicorp.com/terraform/1.3.6/terraform_1.3.6_linux_amd64.zip -o terraform.zip"
      - "unzip terraform.zip -d /usr/local/bin"
      - "chmod 755 /usr/local/bin/terraform"
      - "mv $CODEBUILD_SRC_DIR_TerraformPlan/terraform/$env-terraform.tfplan terraform"
  pre_build:
    commands:
      - "make tf-init"
  build:
    commands:
      - "make tf-apply"

The buildspec file installs terraform, moves the tfplan file back to where it's expected, runs make tf-init (because this is a new container), and then terraform apply

Makefiles

I use Makefiles to simplify the process of deploying both in CodeBuild and when deploying the terraform locally.

The root Makefile for the repo looks like this:

# Copyright 2022 - Chris Farris (chrisf@primeharbor.com) - All Rights Reserved
#
ifndef env
$(error env is not set)
endif

include config.$(env)
export

#
# Terraform
#
tf-init:
    cd terraform && $(MAKE) tf-init

tf-plan:
    cd terraform && $(MAKE) tf-plan

tf-apply:
    cd terraform && $(MAKE) tf-apply

And the Makefile in the terraform directory is:

# Copyright 2022 - Chris Farris (chrisf@primeharbor.com) - All Rights Reserved
#
tf-init:
    terraform init -backend-config=../$(env).tfbackend -reconfigure

tf-plan:
    terraform plan -out=$(env)-terraform.tfplan -no-color

tf-apply:
    terraform apply $(env)-terraform.tfplan

The config.env file contains all the TF_VAR exports to feed variables to terraform similar to:

export TF_VAR_mail_relay_ami=ami-0e03dcd66f...

The $(env).tfbackend contains the line to define the bucket:

bucket="fooli-tf-state-test"

Done!

There you have it, a complete solution to deploy Terraform in CodePipeline with CodeBuild and a manual review of the changes to be made. You can tweak the makefiles and buildspec files as you see fit. Here is the entire CloudFormation Template.

I'm surprised that there isn't a CodeBuild container from Hashicorp or AWS with Terraform pre-installed. The effort of making one would be less than the expense of everyone curling the terraform binary and providers twice on every build.

Mapping your AWS attack surface

Chris Farris — Tue, 27 Dec 2022 02:47:00 +0000

What is a cloud attack surface?

Traditional IT security protects the network perimeter with firewalls, vulnerability scanners and patching processes. Discovering the attack surface of your network perimeter was a simple matter of knowing your public IP range and scanning it from an external host. With the introduction of public cloud technologies, the attack surface expanded to include cloud provider’s APIs. "Identity is the new perimeter" became the new mantra.

The old network perimeter still exists, of course, and it has evolved. No longer are all the IP addresses that make up your perimeter assigned to your company. In the cloud, your network perimeter is part of the cloud provider’s IP space. And, cloud-provider-managed resources provide a direct path to your application code. This combination of cloud-provider IP addresses assigned to you, and the URLs of cloud-provider resources that lead to your application and data, comprise your cloud’s network perimeter and define part of the cloud attack surface.

An Organization must monitor and understand the network perimeter of their cloud estate. Resources comprising the externally facing network components of your cloud attack surface can be broadly grouped into IP addresses, hostnames, and URLs. In this blog post, we will provide step-by-step instructions for mapping the network aspects of the cloud attack surface using Steampipe.

Enumerate IP Addresses using Steampipe

For IP Addresses, we'll look mainly at the Elastic Network Interfaces. While other resources like CloudFront, S3, and API Gateway also leverage IP addresses, the network security protections of those services are the cloud provider's responsibility.

This query will download a list of all public IP addresses tied to the customer’s VPC.

select
  eni.association_public_ip AS public_ip
from
  aws_ec2_network_interface AS eni
where
  eni.association_public_ip is not Null;

Enumerate Hostnames using Steampipe

To determine the DNS Hostnames used as part of your cloud perimeter, Steampipe can query all of the A records and CNAMEs in your Route 53 Hosted Zones. A records point directly to IP addresses under your control. CNAMEs are references that can point to hosts or other cloud-provider-managed resources. In either case, you need to understand what exists in your environment.

To query all of the hostnames that point to CNAME and A Records in your Route 53 Hosted Zones, use this SQL query:

select
  r.name as hostname,
  type,
  jsonb_array_elements_text(records) as resource_record
from
  aws_route53_zone as z,
  aws_route53_record as r
where r.zone_id = z.id
  and (type LIKE 'A' OR type LIKE 'CNAME')
  and z.private_zone=false
  and jsonb_pretty(records) not like '%dkim%'
  and jsonb_pretty(records) not like '%acm-validations.aws.%';

Review the list of IP addresses returned, and if permitted by the terms of service, scan these hostnames for exposed ports and services using nmap or Nessus.

Note: The above query excludes private DNS for VPCs z.private_zone=false and excludes common CNAMEs needed for ACM and email validation.

Enumerate the URLs using Steampipe

A number of AWS services, including CloudFront, S3, API Gateway, and AWS Lambda, produce URLs that can be vulnerable. For example, S3 Buckets exist as URLs on the public internet and can be accessed if the bucket is not properly secured. To get a list of all of the URLs for the public buckets in your cloud environment, you can use this query:

select
  'https://' || name || '.s3.' || region || '.amazonaws.com/' as url
from
  aws_s3_bucket
where
  bucket_policy_is_public is True;

To request all of the URLs from CloudFront, this SQL query will return both the distribution name and any aliases that are part of that distribution:

select
  'https://' || domain_name as url
from
  aws_cloudfront_distribution
UNION ALL
select
  'https://' || jsonb_array_elements_text(aliases -> 'Items') as url
from
  aws_cloudfront_distribution;

API Gateways themselves do not have URLs, but when the customer creates a "stage" a URL is created. To get a list of all the API Gateway v2 URLs, this SQL query can be used:

select 'https://' || api_id || '.execute-api.' || region || '.amazonaws.com/' || stage_name as url
from aws_api_gatewayv2_stage;

API Gateway URLs typically have the following format:

https://[api-id].execute-api.[region].amazonaws.com/[stage]/[path]

where [api-id] is the ID of the API Gateway API, [region] is the AWS region where the API is deployed, [stage] is the stage of the API (such as "prod" or "dev"), and [path] is the path to the specific Lambda function being accessed. For example, the URL "https://abc123.execute-api.us-east-1.amazonaws.com/prod/hello" could be used to invoke a Lambda function named "hello" that is associated with the API Gateway API with ID "abc123", and is deployed in the "us-east-1" region.

Lambda URLs are a new feature of AWS Lambda, released in April of 2022. AWS allows the creation of Lambda URLs that do not require authentication. Therefore, these Lambda URLs are part of your cloud attack surface. To find the Lambda URLs in your environment, you can use this Steampipe query:

select url_config ->> 'FunctionUrl' as url
from aws_lambda_function
where url_config is not Null;

You can pull all these together using SQL UNION queries as demonstrated by this query. You can run it like:

steampipe query all_urls.sql

+-----------------------------------------------------------------------+-------------------------------+
| url                                                                   | type                          |
+-----------------------------------------------------------------------+-------------------------------+
| https://dev-FooliApiStack-780272371.us-east-1.elb.amazonaws.com:443   | application_load_balancer     |
| https://prod-FooliApiStack-1829703886.us-east-1.elb.amazonaws.com:443 | application_load_balancer     |
| https://rwpfqhvj6g.execute-api.us-east-1.amazonaws.com/dev_system     | api_gateway                   |
| https://v5johmngyd.execute-api.us-east-1.amazonaws.com/prod_system    | api_gateway                   |
| https://d7iaso3riah1ur.cloudfront.net                                 | cloudfront_distribution       |
| https://d65z8r284bm5f6.cloudfront.net                                 | cloudfront_distribution       |
| https://da6r321d6ljywr.cloudfront.net                                 | cloudfront_distribution       |
| https://d28s6da8wtxvjm.cloudfront.net                                 | cloudfront_distribution       |
| https://memes.dev1.fooli.media                                        | cloudfront_distribution_alias |
| https://memes.memes.fooli.media                                       | cloudfront_distribution_alias |
| https://oshehk7tpbxfqtkfhco5ojkdyq0fbenx.lambda-url.us-east-1.on.aws/ | lambda_url                    |
| https://f00li-prod-1.s3.us-east-1.amazonaws.com/                      | s3_bucket_url                 |
| https://f00li-dev-1.s3.us-east-1.amazonaws.com/                       | s3_bucket_url                 |
| https://f00li-dev-0.s3.us-east-1.amazonaws.com/                       | s3_bucket_url                 |
| https://f00li-public-bucket-policy.s3.us-east-1.amazonaws.com/        | s3_bucket_url                 |
| https://f00li-prod-0.s3.us-east-1.amazonaws.com/                      | s3_bucket_url                 |
| https://pht-blockchain.s3.eu-central-1.amazonaws.com/                 | s3_bucket_url                 |
+-----------------------------------------------------------------------+-------------------------------+
Time: 115.3s. Rows fetched: 215. Hydrate calls: 317.

Empowered with a list of all the exposed URLs in your organization, you can then set up a process to scan these using a number of web-focused Dynamic Application Security Testing (DAST) tools and scanners such as Zed Attack Proxy, dirsearch (Web path scanner), Aquatone, and Nikto2. The OWASP® Foundation maintains a full list of scanning tools that could be used.

If you are a larger organization that runs a bug bounty program, scanning your URLs for these low-hanging fruit is a quick and easy way to avoid payouts to researchers who use the same tools.

Conclusion

Steampipe is a powerful way to find and enumerate your AWS IP addresses, hostnames, and URLs. Follow the steps outlined here to gather the information needed to define and manage the network component of your cloud attack surface, ensure the security and integrity of your data and applications, and prevent unauthorized access to your network resources. Of course everyone's situation is unique, and you may find a solution that works better for you. If so, please let us know: we love to learn from our community!

Enrich Splunk events with Steampipe

Chris Farris — Tue, 20 Dec 2022 18:27:07 +0000

When analyzing telemetry from AWS in a security operations role, context is key. What is this instance i-7ba5bed288a? Is this random AWS-owned IP address one of mine, or does it belong to someone else? Which account is 178901234562 again? AWS doesn't provide any of this context in CloudTrail or GuardDuty.

If you use Splunk as your Security Event and Incident Management (SEIM) platform, you've probably heard of Lookups. Per Splunk: "Lookups enrich your event data by adding field-value combinations from lookup tables". Lookups can be a great way to improve your detection and investigations by adding attributes and key business context to your CloudTrail, GuardDuty, and VPC Flow Log data.

Steampipe can pull that context into Splunk lookup tables. In a pair of examples, for AWS Accounts and Elastic Network Interfaces, we'll use Steampipe to query AWS Accounts and public and private IP addresses of Elastic Network Interfaces. We'll save that data as Splunk lookup tables. The Steampipe SQL queries and the Splunk SPL queries provided here are examples you can build upon to create your own enrichment tables.

Context enrichment with Steampipe

Steampipe is an open source project that provides a common interface that enables you to query cloud APIs with SQL. WIth Steampipe and the AWS Plugin installed and configured, you can easily run SQL queries against AWS APIs represented as database tables, and export the results to CSV files that load into Splunk as lookup tables.

Let's start with a simple example: a list of all AWS Accounts in an organization. This query (accounts.sql) pulls the twelve-digit account id, Account Name, Status (Active or Suspended), and four specific tags on each account.

select id, name, status,
  tags ->> 'ExecutiveOwner' as Executive_Owner,
  tags ->> 'TechnicalContact' as Technical_Contact,
  tags ->> 'DataClassification' as Data_Classification,
  tags ->> 'environment' as Environment
from aws_payer.aws_organizations_account;

Here's the command to extract this information into a CSV file.

steampipe query accounts.sql --output csv > sp_aws_accounts.csv

The command extracts the account information from AWS Organizations into a CSV file that's ready to be used by Splunk.

id,name,status,executive_owner,technical_contact,data_classification
877426665359,fooli-dev,ACTIVE,Richard Hendricks,Dinesh Chugtai,Public
352894534996,fooli-security,ACTIVE,Bertram Gilfoyle,Bertram Gilfoyle,Internal
152981771857,fooli-prod,ACTIVE,Erlich Bachman,Richard Hendricks ,PersonalInformation
540147993428,fooli-payer,ACTIVE,Monica Hall,Bertram Gilfoyle,None
747037951011,fooli-memefactory,ACTIVE,Erlich Bachman,Richard Hendricks,PersonalInformation
755629548949,fooli-sandbox,ACTIVE,Erlich Bachman,Dinesh Chugtai,None

One of the most useful Splunk lookup tables maps from internal or external IP addresses to the cloud resources they belong to. This Steampipe query gets all of the ENIs in your environment. It joins that data with your account and VPC data to provide account and VPC names. Finally, it populates the attached_resource column with either the EC2 Instance ID or the ENI Description to tell you which resource each public or private IP address belongs to.

select
  eni.network_interface_id,
  eni.private_ip_address,
  eni.vpc_id as vpc_id,
  eni.region,
  eni.status,
  eni.interface_type,
  eni.association_public_ip as public_ip,
  case
    when eni.attached_instance_id is not null
      then eni.attached_instance_id
    else eni.description
  end as attached_resource,
  vpc.tags ->> 'Name' as vpc_name,
  org.name as account_name
from
  aws_ec2_network_interface as eni,
  aws_vpc as vpc,
  aws_payer.aws_organizations_account as org
where vpc.vpc_id = eni.vpc_id
  and org.id = eni.account_id;

Run steampipe query eni.sql --output csv > sp_eni.csv to generate a CSV file like:

network_interface_id,private_ip_address,vpc_id,region,status,interface_type,public_ip,attached_resource,vpc_name,account_name
eni-0bb119c9b8271e13c,10.12.15.127,vpc-0baccbf3534d6a80c,us-east-1,in-use,interface,<null>,arn:aws:ecs:us-east-1:747037951011:attachment/2d6f0e6e-4e26-4dea-ae2f-7f0eac27d471,Prod VPC,fooli-memefactory
eni-03fb47e928c58f8c6,10.12.11.158,vpc-0baccbf3534d6a80c,us-east-1,in-use,interface,52.4.203.49,ELB app/prod-FooliApiStack/564122fa59bf64fe,Prod VPC,fooli-memefactory
eni-019de27cb07fe61f2,10.12.11.162,vpc-0baccbf3534d6a80c,us-east-1,in-use,interface,18.212.183.121,i-0a2af645e985e6aed,Prod VPC,fooli-memefactory
eni-0b0a0f4a74f50324b,10.12.15.233,vpc-0baccbf3534d6a80c,us-east-1,in-use,lambda,<null>,AWS Lambda VPC ENI-prod-FooliMailerStack-mailer-13aa6e45-264b-46cd-a91b-17cc79e7a011,Prod VPC,fooli-memefactory
eni-0d55432870ac355fc,10.12.11.119,vpc-0baccbf3534d6a80c,us-east-1,in-use,interface,54.86.186.254,i-0eeacb7ba5bed288a,Prod VPC,fooli-memefactory
eni-0dc96ad6321864543,10.12.11.151,vpc-0baccbf3534d6a80c,us-east-1,in-use,interface,34.231.250.0,RDSNetworkInterface,Prod VPC,fooli-memefactory
eni-06741026048ceb699,10.12.10.106,vpc-0baccbf3534d6a80c,us-east-1,in-use,interface,34.202.5.187,ELB app/prod-FooliApiStack/564122fa59bf64fe,Prod VPC,fooli-memefactory
eni-03831a874fce92713,10.12.10.104,vpc-0baccbf3534d6a80c,us-east-1,in-use,nat_gateway,44.196.140.136,Interface for NAT Gateway nat-0038ea0ba69861382,Prod VPC,fooli-memefactory
eni-0cb11b13ee1ded995,10.10.11.215,vpc-0c320ebb500ab616a,us-east-1,in-use,interface,3.87.77.3,i-08154eb5935852d50,Dev VPC,fooli-dev
eni-0265d9a496ff2b01e,10.10.10.218,vpc-0c320ebb500ab616a,us-east-1,in-use,interface,44.205.90.161,ELB app/dev-FooliApiStack/6fb5790bfca6ea6f,Dev VPC,fooli-dev
eni-074aac46384fe2501,10.10.10.217,vpc-0c320ebb500ab616a,us-east-1,in-use,nat_gateway,18.206.78.232,Interface for NAT Gateway nat-031b431e19aa13518,Dev VPC,fooli-dev
eni-0f45b8f63195340c5,10.10.11.66,vpc-0c320ebb500ab616a,us-east-1,in-use,interface,54.234.182.17,i-0d1bfdfc785de0619,Dev VPC,fooli-dev
eni-0bf128705f3c36d94,10.8.10.56,vpc-0da7e4ff31018985b,us-east-1,in-use,interface,<null>,i-06944d48a6262f7f9,SecurityVPC,fooli-security

You can now use this lookup table with either the public_ip, private_ip or the network_interface_id.

Upload your file to SplunkWeb or drop the CSV outputs from Steampipe into /opt/splunk/etc/system/lookups on your Splunk search heads, and you're ready to start using these lookups.

How Splunk lookups can simplify your life and help find threats in your environment.

Let's start with a simple example: Who is logging into your AWS accounts via the Web Console? Here we can decorate the ConsoleLogin events with not just the user and IP address, but also include the AWS Account's Name which isn't normally available in CloudTrail.

index="aws_cloudtrail" eventName=ConsoleLogin
| LOOKUP sp_aws_accounts.csv id AS recipientAccountId
  OUTPUT name as account_name
| table account_name, userIdentity.arn, sourceIPAddress

Steampipe-generated lookup tables also work well when you're searching VPC Flow Logs. Here we can cross-reference the FlowLog record dvc (device) with a lookup table of all the network devices to get the names of the resources that this target IP address was talking to.

index=aws_flowlogs 3.236.91.34
| LOOKUP sp_eni.csv network_interface_id AS dvc
  OUTPUT attached_resource as attached_resource, vpc_name as VPC, account_name as Account
| stats count by attached_resource, VPC, Account

One final example: Using ENI data to cross-reference where an EC2 Instance Role is coming from:

index=aws_cloudtrail  "userIdentity.arn"="arn:aws:sts::*:assumed-role/payments_role/*"
| LOOKUP sp_eni.csv public_ip AS sourceIPAddress
  OUTPUTNEW attached_resource as attached_resource
| fillnull attached_resource value="External IP"
| stats count by sourceIPAddress, eventName, attached_resource

If a role used by an EC2 Instance is generating CloudTrail events from an IP address not part of your VPC, that's a major concern that needs to be investigated. Here we see a role that belongs to an EC2 Instance being used outside from IP Address that are not tied to known EC2 Instances.

Extend your enrichment activities with Steampipe

AWS Accounts and elastic IP addresses are only two examples of event enrichment for your SEIM. If you have a tagging strategy for your EC2 Instances, you can query instance names and contact teams to correlate them with your endpoint detection and response (EDR) alerts.

Your organization is probably polycloud, and so is Steampipe. Plugins exist for all the major cloud providers; you can easily decorate your Azure activity logs and GCP audit logs.

The sample queries here were only examples of the power of Steampipe to enrich logging data. If you try this technique, let us know. We love hearing how practitioners use Steampipe.

Can't miss Security Sessions at re:Invent 2022

Chris Farris — Mon, 10 Oct 2022 12:35:06 +0000

Session registration for re:Invent 2022 opens in a few days, and Luc van Donkersgoed inspired me with his post to highlight what you can't miss in the Security, Compliance, and other cool topics.

The session planner doesn't let me deep-link, so I'll be including the session numbers. I won't include time and place since I expect those to change. The description from the session catalog is included.

Can't miss security sessions

SEC401 - AWS Identity and Access Management (IAM) policy evaluation in action (Workshop)

One of the most popular and talked about chalk talks from re:Inforce was Matt Luttrell and Dan Peebles diving deep into policy evaluation. They provided new diagrams and animations to explain how SCPs, permission boundaries, and resource policies all interact with identity-based policies, conditions, and different effects. Matt is going to reprise and extend this work into a two-hour workshop. This session is a must if you're fighting SCPs, Permission Boundaries, and Resource Policies.

In this workshop, dive deep into the logic of AWS Identity and Access Management (IAM) policy evaluation. Gain experience with hands-on labs that walk through IAM use cases and learn how different policies interact with each other. Using identity- and resource-based policies within single- and cross-account scenarios, learn about the evaluation logic that you can apply in your own environment. You must bring your laptop to participate.

SEC402 - The anatomy of a ransomware event targeting data residing in Amazon S3 (Chalk Talk)

This is a repeat of the chalk talk given at re:Inforce and was one of the best sessions from that event. Kyle works on the Customer Incident Response Team and will dive into real-world cases that AWS has seen in the field of Ransomware attacks on customers. You'll come away from this session with action items based on actual threat intel.

Ransomware events can cost governments, nonprofits, and businesses billions of dollars and interrupt operations. Early detection and automated responses are important steps that can limit your organization’s exposure. In this chalk talk, walk through the anatomy of a ransomware event that targets data residing in Amazon S3 and hear detailed best practices for detection, response, recovery, and protection.

BOA204 - When security, safety, and urgency all matter: Handling Log4Shell (Breakout Session)

The indomitable Abby Fuller will talk through how AWS internally handled Log4Shell. How a $1.6T company handled Log4Shell and how you handled Log4Shell are not really comparable, I think this session will have a lot to offer for the next time something like this hits. It's also in that quiet, sweet spot of Friday morning when nothing else is happening and everyone is wrapping up.

On December 9, 2021, there was a report of a potential remote code execution issue in the widely used open-source Apache logging library Log4j. This issue allowed a user to use Java Naming and Directory Interface (JNDI) and LDAP endpoints to execute arbitrary code on a system. Over the next 10 days, 5 additional common vulnerabilities and exposures affecting Log4j were made public. This event as is now referred to as Log4Shell. In this session, learn about the response to Log4Shell, from initial notification to hot patch, fleet scanning, and customer communications.

COP305 - Best practices for organizing and operating on AWS (Breakout Session)

Full Disclosure: The customer speaker here is my former boss, and she's talking about the AWS Organizations governance work my old team and I did while I was at Discovery. Learn how we tamed 200+ AWS accounts and balanced the needs of security, finance, and the whims of half a dozen different business units.

Managing and operating cloud environments from multiple business units can be challenging. In this session, hear from Bianca Lankford, Senior Director/Global Head of Cloud Engineering & Governance from Warner Brothers Discovery, how they organized their cloud environment to allow teams to develop with agility while being able to manage and operate their applications in a secure, automated, reliable, and cost-effective way. See how you can use AWS Organizations and AWS Systems Manager to operate your applications at scale, manage mergers and acquisitions, and develop governance as a product for your environment.

SEC206 - Security operations metrics that matter (Chalk Talk)

One of the things I never finished at my last job was building out cloud security key risk indicators (KRIs) for my program. Anna was one of my partners in crime for my adversary emulation chalk talk at last year's re:Invent, so I'm keen to hear what she's built for her customers.

Security tooling can produce thousands of security findings to act on. But what are the most important items and metrics to focus on? In this chalk talk, learn about a framework you can use to develop and implement security operations metrics in order to prioritize the highest-risk issues across your AWS environment. This includes applying critical business context and capturing the metrics across your multi-account environment so you can take action with confidence.

SEC330 - Harness the power of temporary credentials with IAM Roles Anywhere (Chalk Talk)

I'll be honest. I have not yet kicked the tires on IAM Roles Anywhere. This service promises to eliminate the need for IAM Users, but the key management issues make me wonder how many enterprises will do it right. This is a chalk talk, so I hope for some lively discussion and to discover some of the risks and edge cases with this new service.

In this chalk talk, get an introduction to AWS Identity and Access Management (IAM) Roles Anywhere, and dive deep into how you can use IAM Roles Anywhere to access AWS services from outside of AWS. Learn how IAM Roles Anywhere securely delivers temporary AWS credentials to your workloads. Discover the different use cases that IAM Roles Anywhere is designed to address as well as best practices for using it.

COP323 - Delegating access in a multi-account environment with IAM Identity Center (Chalk Talk)

AWS SSO (I refuse to use the new name) has changed a lot since I first deployed it for the SECCDC. We started using it at my last job, and I want to see the additional capabilities they introduced.

In this chalk talk, learn about delegating access management with AWS Organizations and AWS Control Tower using AWS IAM Identity Center. Using customer-managed policies and permissions boundaries, you can enable a decentralized access management model with permissions guardrails that enforce coarse-grained authorization standards that apply in both role-based and attribute-based access control (RBAC and ABAC) models.

MKT304 - Faster vendor risk assessments with AWS Marketplace Vendor Insights (Chalk Talk)

One of the more under-appreciated announcements from AWS re:Inforce was AWS Marketplace Vendor Insights, or as I thought of it at the time Simplify Vendor Risk Management as a Service. I'm hoping this chalk talk will help me understand the capabilities of this new service and how security practitioners can use it to make their lives easier.

Validating the compliance and security posture for third-party SaaS products is often a complex process. In this chalk talk, explore how AWS Marketplace Vendor Insights simplifies the SaaS risk assessment process to help enterprises procure trusted software. Built upon AWS Config, AWS Audit Manager, and AWS Artifact, Vendor Insights streamlines the process by providing on-demand access to security and compliance information via a web-based dashboard. Learn how vendors can provide customers with on-demand access to security and compliance information to showcase security and compliance excellence.

SEC321 - Building your forensics capabilities on AWS (Chalk Talk)

I've spent a lot of time recently giving talks and classes on Incident Response in AWS, and EC2 Forensics is still a topic that lacks a definitive way of doing it. I look forward to Jonathon Poling's discussion of the topic in this chalk talk.

You have a compromised resource on AWS. How do you acquire evidence and artifacts? Where do you transfer the data, and how do you store it? How do you analyze it safely within an isolated environment? Join this chalk talk to walk through building a forensics lab on AWS, methods for implementing effective data acquisition and analysis, and how to make sure you are getting the most out of your investigations. Learn how to identify the tools and capabilities you need to effectively analyze it, as well as how to maintain least-privilege access to the evidence. Finally, walk through how to learn from your analysis and investigation to improve your security.

SEC202 - Vulnerability management with Amazon Inspector and AWS Systems Manager (Builder's Session)

Vulnerability Management sucks. I hate it. I'd rather farm alpaca than work in the VM space as it's implemented in most enterprises. That said, I've got ideas on how to improve it for cloud workloads (spoiler alert: that was one of the reasons I went to my current job). I've not done a builder's session before, and it's only a 200-level session, but I know one of the builders, so I'll be signing up for this one.

Join this builders’ session to learn how to use Amazon Inspector and AWS Systems Manager Patch Manager to scan and patch software vulnerabilities on Amazon EC2 instances. Walk through how to understand, prioritize, suppress, and patch vulnerabilities using AWS security services. You must bring your laptop to participate.

SEC208 - Executive security simulation (Workshop)

I've not done much with tabletop exercises, and I've not yet done one focused on Cloud Security, so I think this one is a better use of my time than cheap beer and big crowds on the expo floor.

This workshop features an executive security simulation, designed to take senior security management and IT or business executive teams through an experiential exercise that illuminates key decision points for a successful and secure cloud journey. During this team-based, game-like simulation, use an industry case study to make strategic security, risk, and compliance decisions and investments. Experience the impact of these investments and decisions on the critical aspects of your secure cloud adoption. Learn about the major success factors that impact security, risk, and compliance in the cloud and applicable decision and investment approaches to specific secure cloud adoption journeys. You must bring your laptop to participate.

SEC329 - AWS security services for container threat detection (Breakout Session)

I will admit I'm not an expert on container security. My personal development path jumped from EC2 straight to Lambda functions. So, I'm interested in a deeper dive into what my former colleague Mrunal did with the topic.

Containers are a cornerstone of many AWS customers’ application modernization strategies. The increased dependence on containers in production environments requires threat detection that is designed for container workloads. To help meet the container security and visibility needs of security and DevOps teams, new container-specific security capabilities have recently been added to Amazon GuardDuty, Amazon Inspector, and Amazon Detective. In this session, learn about these new capabilities and the deployment and operationalization best practices that can help you scale your AWS container workloads. Additionally, the head of cloud security at HBO Max shares container security monitoring best practices.

Sessions highlighting the massive awesomeness of AWS

SEC404 - A day in the life of a billion requests (Breakout Session)

Eric Brandwine will dive into the scale and operational considerations of IAM (ignore the ironically bad session number). Let's face it: there is nothing actionable here for you. This is just a chance to nerd-out on a cool topic.

Every day, sites around the world authenticate their callers. That is, they verify cryptographically that the requests are actually coming from who they claim to come from. In this session, learn about unique AWS requirements for scale and security that have led to some interesting and innovative solutions to this need. How did solutions evolve as AWS scaled multiple orders of magnitude and spread into many AWS Regions around the globe? Hear about some of the recent enhancements that have been launched to support new AWS features, and walk through some of the mechanisms that help ensure that AWS systems operate with minimal privileges

SEC327 - Zero-privilege operations: Running services without access to data (Breakout Session)

If you recall the Twitter compromise in 2020, or the more recent Okta incident, both had the same things in common: Employees had high levels of access to production and customer data. In this session Colm MacCarthaigh will discuss how AWS keeps its employees out of your data. Who knows, I may even change my opinion on Encryption in AWS.

AWS works with organizations and regulators to host some of the most sensitive workloads in industry and government. In this session, learn how AWS secures data, even from trusted AWS operators and services. Explore the AWS Nitro System and how it provides confidential computing and a trusted runtime environment, and dive deep into the cryptographic chains of custody that are built into AWS Identity and Access Management (IAM). Finally, hear how encryption is used to provide defense in depth and why we focus on verified isolation and customer transparency at AWS.

ARC310 - Beyond five 9s: Lessons from our highest available data planes (Breakout Session)

Another session from Colm MacCarthaigh on the scale and reliability of AWS Services. If I can't make this in-person, it will be at the top of my viewing list when I get home.

Updated with recent learning, this session dives deep into building and improvising resilience in AWS services. Every AWS service is designed to be highly available, but a small number of what are called Tier 0 services get extra-special attention. In this session, hear lessons from how AWS has built and architected Amazon Route 53 and the AWS authentication system to help them survive cataclysmic failures, enormous load increases, and more. Learn about the AWS approach to redundancy and resilience at the infrastructure, software, and team levels and how the teams tasked with keeping the internet running manage themselves and keep up with the pace of change that AWS customers demand.

Other Sessions I'm going to

STG215 - Unlocking business value in media and entertainment with Amazon S3 (Breakout Session)

2016 was the year of the spreadsheet for me. I was designing the financial model for moving the (then 14PB) CNN Library into AWS S3. Glacier Expedited Retrieval wasn't a thing then, so it was lots of "how can we use S3 IA". I'm excited to see how this project has moved forward with former colleague Jay Brown talking about how they're now using S3.

Media and entertainment companies are creating more content than ever to engage audiences and grow revenue, but many are overlooking the hidden value of content locked in their media archives. The proliferation of screens connected to the internet provides customers with more ways of consuming content whenever and wherever. Between the influx of choice and ubiquitous connectivity, storing media content is challenged to keep up with not only growth in storage but also capabilities needed to support this multiscreen world. In this session, Amazon S3 customers Warner Bros., CNN, and PGA Tour share how migrating media archives from on-premises systems to the cloud can unlock business value for your organization.

DOP402 - Implementing DevSecOps pipelines with compliance in mind (Chalk Talk)

I'll admit I don't know enough about CI/CD, and I don't practice what the cloud security community preaches in this regard. I can't pass up a 400-level chalk talk on a subject I need to do more with.

In this chalk talk, review a DevSecOps CI/CD pipeline that includes software composition analysis, static application security testing, and dynamic application security testing. Also learn about best practices for incorporating security checkpoints across various pipeline stages and aggregating vulnerability findings into a single pane of glass. Finally, hear about processes and tools that can increase an organization’s ability to deliver applications and services in a secure manner.

COP325 - Migrate AWS accounts like an expert (Chalk Talk)

While this isn't my job anymore, I spent a lot of time thinking through this process, and I'm curious about what AWS will suggest here.

This chalk talk is for you if you’ve ever had to migrate an AWS account from one organization to another during a merger and acquisition activity, while migrating to AWS Control Tower, or just while organizing your AWS environment according to best practices. In this talk, learn how to identify dependencies in your current organization that you can proactively address before the migration. The talk covers code for detecting resource policies and identity policies with dependencies. Walk through additional checks that can help you achieve a quick and efficient migration.

STG404 - Explore Amazon EBS direct APIs with flexible snapshot proxy (Chalk Talk)

Until recently, the only way to get EBS data out of the cloud was to mount it to a machine and run dd | ssh. The EBS Direct APIs are new to me, and there are some interesting security implications with these capabilities I want to know more about.

Amazon EBS snapshots are a feature-rich data protection function used by enterprises for block-level data. Join this chalk talk to learn how flexible snapshot proxy, an open-source project that uses Amazon EBS direct APIs, can enable you to efficiently move data between applications in a cross-Region, cross-organization, logically air-gapped replication scenario without temporary copies. Understand how a block of data moves through systems, services, and geographies. Also learn how to eliminate temporary copies, reduce transfer costs, improve RTO/RPO, and integrate your on-premises applications and systems with Amazon EBS. This talk dives into field-proven architectural patterns for building global-scale real-world solutions.

re:Play

I joke that my annual pilgrimage to re:Invent is my "Cloud Nerd Rave in the Desert". Thursday night's re:Play is that rave. Even if you hate crowds and loud music, you should attend once to see the spectacle and realize that Frugality is a selective leadership principle.

Open Source Zone and Steampipe

I didn't pitch a talk for re:Invent this year, but I will be presenting. Come by the Open Source Zone (Third floor of the Venetian near San Polo and the Press area) on Tuesday from 1pm to 3pm and I'll be demoing our open source tool Steampipe and a number of the nifty things it can do to help manage your cloud sprawl and reduce risk in your organization.

Pre-registration

Pre-Registration opens on October 11th at 1pm EDT for a guaranteed seat in these sessions. I'm told that 25% capacity is reserved for walk-ups, and in the past there have been a number of no-shows. I've also been denied entrance to a colleague's presentation because they ran out of open seats, and AWS policy doesn't allow anyone to stand in the back. So may the odds be ever in your favor.