Why Removing a AWS Security Group Rule Does Not Stop Traffic Flow

Here’s the situation.  I created an AWS Security Group inbound rule to allow Ping, which allows me to ping an EC2 instance from my home computer. The issue is that when I remove the Ping inbound rule, Ping requests are still receiving a response.  I expect that when I remove the Ping inbound rule, the responses should stop.

In this post, I will reproduce the issue and show you that this is a feature of AWS Security Groups and not a bug.

Initial Setup:

• 1 VPC 192.168.0.0 /16
• 1 AZ
• 1 EC2 Instance with public IP – 54.245.15.244
• Internet Gateway (IGW)
• Route
  • 0.0.0.0/0 -> igw-a664ec1
  • All traffic can pass through IGW
  • 192.168.0.0/16 -> local
  • All resources can talk to other resources within the IP Range
• Security Group assigned to EC2 Instance is empty

There are no inbound Rules for Security Group WBC-Web.  No traffic will be permitted

Even though this EC2 instance has a Public IP, no inbound traffic should reach this instance since this instance is associated the Security Group WBC-Web with no Inbound Rules.

Let’s Validate this Setup

From my home computer, if I ping the EC2 instance, I will receive a “request time out.” message

There is no response since the Security Group has no inbound rules.

Create Inbound Ping Rules

In AWS console, I added the inbound rule to Security Group WBC-Web to allow ping from all IP addresses.

Now if I try to ping the EC2 instance, I will receive a response.

The Confusion

If I remove the Security Group inbound ping rule, I continue to receive a response, which I do not expect. I expect the ping should timeout.

To demonstrate this, I will run ping with the -t option. This will keep ping running until I cancel it manually.

I will start the ping before I remove the ping rule.

In the image below, the red line marks the point when I removed the ping inbound rule.  After I remove the ping inbound rule, a response is still returned.

I stopped the ping and started the ping to see if I would still receive a response, which I did receive a response.

I stop the ping again and waited a minute before starting it again. This time the ping finally timed out.

So What’s Going On

For Security Groups there is a feature called Connection Tracking.  Connection Tracking is used to track information to and from an instance.

When there’s a flow of traffic, for our example a ping, and the security group inbound rule is removed, the flow of traffic will not be stopped by the removal of the inbound rule.  The traffic is interrupted when it’s stopped on the client or the host for at least a few minutes.

Removing the ping inbound rule while traffic is flowing will not interrupt or stop the flow of traffic. You must first stop the ping on the client or host and wait at least 1 minute before the removal of the inbound rule will take affect.

I believe Amazon does a better job of describing this.

Not all flows of traffic are tracked. If a security group rule permits TCP or UDP flows for all traffic (0.0.0.0/0) and there is a corresponding rule in the other direction that permits all response traffic (0.0.0.0/0) for all ports (0-65535), then that flow of traffic is not tracked. The response traffic is therefore allowed to flow based on the inbound or outbound rule that permits the response traffic, and not on tracking information.

TCP traffic on port 22 (SSH) to and from the instance is tracked, because the inbound rule allows traffic from 203.0.113.1/32 only, and not all IP addresses (0.0.0.0/0). TCP traffic on port 80 (HTTP) to and from the instance is not tracked, because both the inbound and outbound rules allow all traffic (0.0.0.0/0). ICMP traffic is always tracked, regardless of rules.

An existing flow of traffic that is tracked may not be interrupted when you remove the security group rule that enables that flow. Instead, the flow is interrupted when it’s stopped by you or the other host for at least a few minutes (or up to 5 days for established TCP connections). For UDP, this may require terminating actions on the remote side of the flow. An untracked flow of traffic is immediately interrupted if the rule that enables the flow is removed or modified. For example, if you remove a rule that allows all inbound SSH traffic to the instance, then your existing SSH connections to the instance are immediately dropped.

Security Group – Connection Tracking

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#security-group-connection-tracking

It’s important to understand that if the traffic is being tracked and is flowing then removing an allow rule will not stop the traffic.

The following are a few way to stop the traffic.

1. Create a Network Access Control List (ACL) inbound rule
2. On the instance, disable the communication type or block the port
3. Delete the Security Group

While learning about AWS Security Groups, I ran into this situation and was very confused. It took a lot of digging to understand what was going on.  I hope that this will help others and that others will not go through the pain I went through.

Update – Saturday, Nov 11, 2017

This does not seem relevavent to SSH connections.  As soon as I remove an SSH inbound rule the SSH connection is terminated.  I assume SSH and RDP rules would function similarly.

 

The post Why Removing a AWS Security Group Rule Does Not Stop Traffic Flow appeared first on Mike Barlow (BarDev).