AWS Incident Response: How To Contain An EC2 Instance?

Introduction

In a world where it is a matter of "when" an incident would take place, all organizations must keep a well-thought-out plan to act upon identifying an incident that would possibly cause monetary damage as well as damage to the organization's image.

The humble EC2 instance provided by AWS has quickly become one of AWS's most sought compute services. However, with its wide use, it is equally essential to ensure that administrators know the correct steps to minimize the damage when responding to an EC2 instance.

What Is Incident Response?

Incident response is a structured approach that allows organizations to prepare, detect and respond to security incidents promptly while reducing the impact these incidents may have on the organization.

Understanding and implementing the measures to accommodate for incident response allows organizations to have documented procedures and adequate technical controls to conduct investigations and contain the infection without allowing it to spread across the organizations.

An Incident Response Plan is at the heart of any well-organized incident response as it documents the organization's procedures and responsibilities of each member within the incident response team.

AWS Shared Responsibility Model

Most organizations using cloud infrastructure has a fundamental misconception that security and maintenance become the responsibility of a cloud service provider such as AWS.

However, this is not true since AWS marks the responsibility boundaries of the different components that AWS and the customer are responsible for within its shared responsibility model.

Understanding the shared responsibility model is crucial to determine the controls and responsibilities that need to be undertaken by the customer/user to manage, secure and maintain the cloud services effectively.

Reference: https://aws.amazon.com/compliance/shared-responsibility-model/

According to the AWS shared responsibility model, it is the responsibility of AWS to provide and maintain the following components:

1. Hardware and infrastructure running the global AWS services
2. Software that facilitates the compute, storage, database and networking of the global AWS services

Within the constraints of the AWS EC2 service, the shared responsibility model states that it is the responsibility of the customer/user to configure and maintain the following components:

1. Encryption of traffic and data
2. Operating system, network and any firewall-related configurations
3. Applications and IAM
4. Data hosted on AWS and EC2 instances

It also means that incident response and other security measures related to the operating system or any instance created on AWS must be provided and maintained by the customer.

Incident Response Lifecycle

The NIST incident handling guide highlights the stages that must be looked at during an incident response. These stages entail the different steps necessary to protect resources from attackers and respond to an incident when a resource is compromised.

The typical incident response lifecycle contains the following stages:

1. Planning
2. Detection
3. Analysis
4. Containment
5. Eradication
6. Recovery

Each stage holds a vital function within the incident response lifecycle, ensuring that an organization can best prepare for an incident.

Stage 01: Planning

This stage of the incident response lifecycle associates the necessary planning of documentation and other technical controls.

One of the most critical documents covered during this phase is the incident response playbooks; these playbooks have step-by-step guides and responsibilities mapped out for the entire incident response team. This structured approach ensures that incident response teams are aware of their responsibilities and the approach is taken to respond to a specific type of incident such as malware infection, C2C communication, etc.

This stage will also cover the planning of the necessary technical control to support the protection and incident response of the resources within an organization.

Stage 02 - 03: Detection & Analysis

Detection of malicious activities within the AWS infrastructure could vary depending on the type of monitoring required.

A popular method of monitoring malicious activity within an EC2 instance is using a third-party EDR or XDR solution. However, this requires a separate license and other requirements set by the product manufacturer.

Another popular method of gaining detailed insight into the AWS infrastructure and EC2 instances is a SIEM (Security Information and Event Management) solution that allows collecting and correlating multiple log sources to detect anomalies within the AWS resources.

AWS also provides native solutions such as Amazon GuardDuty and Amazon Inspector to detect and respond to threats effectively by allowing automated responses.

Stage 04: Containment

Containment aims to limit the reachability of an infected resource so that it may not cause any further harm to any other resource within the AWS infrastructure.

In this case, isolating an EC2 instance would provide adequate time for the incident response team to analyze and determine the scope of the infection. It also allows forensic investigators to conduct forensic analysis to determine more detailed information required during the later stages of the incident response lifecycle.

The first approach that most users may take when isolating an EC2 instance is to restrict the network connectivity originating to and from the infected EC2 instance; this ensures that the attackers cannot maintain any existing backdoor communications or establish any further connections. It also ensures that no user or application has access to the infected EC2 instance that could taint any evidence within the EC2 instance or allow it to infect other resources within the AWS infrastructure further.

This article focuses on exploring the different approaches to isolating an EC2 instance.

How to contain an EC2 instance?

Users can use multiple approaches when isolating an EC2 instance. The approach used will depend on the level of isolation required.

The most common methods of isolation are:

1. Security Groups
2. NACLs
3. Route Tables
4. Internet Gateways

Containing an EC2 Using Security Groups

Security groups are often the most common approach when isolating an EC2 instance due to their ease of use and targeting isolation capabilities.

A security group's default "implicit deny" behaviour makes it a suitable isolation method since a security group without rules will deny all traffic; a security group requires a user to specify rules to allow specific traffic.

However, it is essential to note that adding a security group to an EC2 instance will not restrict traffic, and AWS will grant access to the security group with the most permissive rules.

Therefore to restrict access by using a security group, users must remove all existing security groups from an EC2 instance or remove all the rules from the existing security group.

When using security groups for isolation, it is crucial to consider the connections an EC2 instance may establish via a security group. These types of connections are:

1. Untracked Connections
2. Tracked Connections

Untracked Connections

Untracked connections are any connection established from a rule with a "0.0.0.0/0" wildcard. The direction of these connections can be both inbound and outbound.

The significance of these types of connections is that any change to a security group, such as a rule change or security group change, will immediately affect the traffic flow.

Tracked Connections

Tracked connections are any connection established from a rule with a specific IP address or segment, such as rules that allow traffic from "43.101.53.202/32". The direction of these connections can be both inbound and outbound.

Unlike untracked connections, changing a rule within a security group or deleting a security group from an EC2 instance will not immediately affect the traffic flow. This property of tracked connections may allow an attacker to maintain established connections even after the specific rules are removed.

Isolating EC2 instances with Tracked Connections

Users must follow the steps mentioned below to isolate an EC2 instance with a tracked connection:

1. Create a dedicated isolation security group without any rules
2. Create a single rule of 0.0.0.0/0 for all traffic in both the inbound and outbound rules
3. Remove any existing security groups attached to the EC2 instance
4. Associate the newly created isolation security group to the instance
5. Delete both the inbound and outbound rules you created for the isolation security group

Following these steps will effectively convert all the existing tracked connections into untracked connections, thus terminating any existing connections.

Planning Ahead With Security Groups

Each second could be the difference between successful isolation or further infection in a time-sensitive operation such as an incident response. Therefore users may keep preconfigured security groups that can be used to perform EC2 isolation.

a. Create a “Step 01” security group with the following rules

b. Create a "Step 02" security group without any rules

After creating these two security groups, users will now be able to swiftly isolate an EC2 instance while eliminating any tracked connections by using the following steps:

1. Remove all existing security groups or rules within existing security groups
2. Add the “Step 1” security group to the EC2 instance
3. Remove the “Step 1” security group from the EC2 instance
4. Associate the “Step 2” security group to the EC2 instance

Containing an EC2 Using NACLs

NACLs or Network ACLs is the next approach users may use to isolate an EC2 instance. However, NACLs are not targeted; therefore, it is difficult to isolate an individual EC2 instance using an NACL.

A Network ACL relies on rules to allow and deny traffic from subnets; however, unlike security groups, NACLs are stateless, meaning that rules need to be defined for both the request and the response of each connection.

The most significant advantage of using a NACL is that there is no multistage process compared to a security group. A single inbound or outbound rule will deny all traffic towards the respective direction.

However, since NACLs rely on rules based on subnets. Therefore, using NACLs to isolate an EC2 instance will affect all the other instances within the specified subnet.

Users may choose to use an existing NACL for isolation. In this method, users are required to add the deny rule as the first rule of the NACL; if the NACL is full existing rules may need to be removed to make space for the isolation-specific rules at the top of the NACL.

The other approach is to create a new NACL with the deny rules at the start of the NACL and to associate the dedicated isolation NACL to the VPC that hosts the affected EC2 instance.

Containing an EC2 Using Route Tables

Users can use route tables to isolate EC2 instances to move up the isolation spectrum. Route tables are an excellent method of isolating EC2 instances from all external networks, such as Internet Gateways, Direct Connections and VPN connections.

However, it is essential to understand that the route table is associated with the VPC that hosts the EC2 instance and not the EC2 instance itself. Therefore even though using a route table for isolation terminates all connections to all external networks, it will also disrupt any connection flowing through the specific VPC.

Additionally, isolating EC2 instances using route tables will restrict external access to the affected EC2 instance. However, it will still be able to communicate within the subnet and continue to spread to other resources within the same subnet.

Containing EC2 Using Internet Gateways

EC2 Isolation using internet gateways restricts internet connectivity to the VPCs that host the EC2 instances. However, restricting access using internet gateways are not straightforward since AWS will not allow users to remove the internet gateways if any EC2 dependencies within the VPC require the internet gateway.

Users are required to remove all dependencies to isolate an EC2 instance using an internet gateway successfully. However, this is not feasible during an incident response scenario.

Instead, users may obtain the same effect by removing all the internet gateway routes from the routing table or attaching a custom route table with no rules for all subnets.

EC2 Containment Playbook

A well-balanced and practical incident response playbook must cover all possible aspects when containing an affected resource. It must not restrict itself to a network containment approach but explore all possible methods of limiting access to and from the affected EC2 instance.

The following playbook provides a good starting point for organizations to adopt an EC2 containment playbook that covers most basic containment areas.

1. Tag and detach the EC2 instance from any auto-scaling group
2. Create a new security group that denies both inbound and outbound traffic (Empty Security Group)
3. Remove any existing security groups and associate them with the “empty” security group
4. Remove any IAM roles associated with the instance
5. Create a snapshot of the root volume
6. Create an AMI (Amazon Machine Image) of the instance for later analysis

It is also important to tag all entities used for isolation so that the same resources will not be used for any other business operation that can render the isolation ineffective.

Key Takeaways

This article explains the various approaches available for EC2 instance isolation. However, using a single method for isolation may not be sufficient since other factors may force a user to select a combination of these isolation approaches.

Additionally, network containment is only one part of a comprehensive isolation plan. Therefore, users must isolate other aspects, such as additional roles, to ensure comprehensive isolation.

I hope you have found this helpful. Thank you for reading!