DEV Community: Andy Lim

UFW and Docker Security on Linux machine

Andy Lim — Fri, 27 Aug 2021 08:53:11 +0000

Uncomplicated Firewall (UFW) is served as a frontend tool for iptables in Linux-based machine. It's used to provide easy interface to manage firewall so you don't have to manage them through complicated iptables and netfilter commands.

So today I tried experimenting UFW with NGINX running as a Docker container on Amazon Linux 2 EC2 instance in my AWS environments.

Before this, I have launched an Amazon Linux 2 EC2 instance and have SSH and TCP port 80 allowed for incoming request in the security group attached to the instance.

Install UFW

First, ssh into the EC2 instance.



$ ssh -i example.pem ec2-user@1.23.45.678

By default, UFW is not pre-installed on Amazon Linux 2 and not available in RHEL repository. We will need to install EPEL repository to our system.



$ sudo yum update -y 
$ sudo yum install epel-release -y

Then, install UFW by running command below:



$ sudo yum install --enablerepo="epel" ufw -y

Then, start UFW service and enable it to start on boot time.



$ sudo ufw enable

Run command below to ensure UFW is running:



$ sudo ufw status

Install Docker

Next, install Docker runtime in the instance:



$ sudo amazon-linux-extras install docker
$ sudo service docker start
$ sudo usermod -a -G docker ec2-user

Confirm Docker is installed and you see version as output:



$ sudo docker -v

Docker version 20.10.7, build f0df350

Then, run a NGINX container using command below:



$ sudo docker run -d --name my-nginx -p 80:80 nginx

This command will pull the Docker image from Docker Hub and run as a container on local Docker runtime. Run command below to see if my-nginx container is running:



$ sudo docker ps

CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                               NAMES
c3aee3db60f1   nginx     "/docker-entrypoint.…"   22 seconds ago   Up 21 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp   my-nginx

Next, run curl command to see if your website is running perfect or go to your browser and search.



$ curl -L 1.23.45.678:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

You will see a beautiful NGINX website in front of you!

Problem arises

Next, here is something fun: I want to deny anyone looking for this website through UFW!

Go back to your EC2 instance. Run the command below to enable the rule of denying incoming request with port 80 in UFW:



$ sudo ufw deny 80


Rule updated
Rule updated (v6)

Validate the deny rule is enabled:



$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
SSH                        ALLOW       Anywhere
224.0.0.251 mDNS           ALLOW       Anywhere
22                         ALLOW       Anywhere
80                         DENY        Anywhere
SSH (v6)                   ALLOW       Anywhere (v6)
ff02::fb mDNS              ALLOW       Anywhere (v6)
22 (v6)                    ALLOW       Anywhere (v6)
80 (v6)                    DENY        Anywhere (v6)

Now, test on your host machine and check again. You should not be able to see the NGINX website on your host machine!



$ curl -L 1.23.45.678:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Why are you still here.....

What's the problem and how to fix it

From Docker documentation, I only learned that by default, Docker directly manipulates iptables for network isolation.

There are several ways to disable this. One of it is that we don't expose the port 80.

Another way is, we can disable this Docker behaviour by creating or modifying /etc/docker/daemon.json.



$ sudo vi /etc/docker/daemon.json

{ "iptables": false }

$ sudo service docker restart

By this, we don't have to worry if other port is exposed and open when we run another container with different ports exposed.

Last, try to connect again:



$ curl -L 1.23.45.678:80

curl: (7) Failed to connect to 1.23.45.678 port 80: Operation timed out

Done!

Conclusion

This article may not explain well about the fundamental of Linux security and you may think why do I need UFW when there's security group attached on EC2 instance and you can configure the allow/deny rules easily from security group.

Well, it's for experiment purpose! I never learned about UFW or security related topics in Linux and I thought security group is the only solution to secure my instances.

Let me know in the comments about your thoughts too!

Happy coding! 💻

The Art of Thinking Clearly - Book Review

Andy Lim — Mon, 12 Apr 2021 04:18:19 +0000

Introduction

Recently I set a goal for myself to read up to 10 pages of a book everyday. I know it may sound easy for some people, but for someone who doesn't really like to read, like me, it's just like asking someone to play F chord when one doesn't know how to play guitar. This is definitely a consistent breakthough and although I missed out to reading (average 1 day per week), but I managed to maintain the consistency, just like coding. I would like to share my book review about a book that I read recently, The Art of Thinking Clearly by Rolf Dobelli.

What's so special about this book

This book covers the topic of cognitive biases which each cognitive bias is separated into one chapter and there are 99 chapters in this book. For those who don't know, cognitive bias is a type of error in thinking that occurs when people are processing and interpreting information in the world around them. It is often a result of your brain's attempt to simplify information processing and it affects the way we see and think about the world.

How does it relate to my career journey

There are several cognitive biases that catch into my eyes and find it relatable to my career.

Swimmer's body illusion

This cognitive bias describes how we confuse traits with result. We think that we can get the body of a professional swimmer in Olympia level by swimming a lot, but the truth is, they are professional swimmer because of their good physiques. Similarly, the female models advertise cosmetics to attract female customers to believe that these products make them beautiful. But it's not the cosmetics that make these women model-like but simply those models are born attractive. How their bodies are designed is a factor of selection and not the results of their activities.

I thought that I can be smart by coding a lot like a professional software developer, but the truth is, they are good because they are smart. Being smart is one of traits to be a good and professional software developer, but not the result.

Déformation professionnelle

This cognitive bias describes a tendency to look at things from the point of view of one's own profession or special expertise, rather than from a broader or humane perspective. This book states a good insight, If your only tool is a hammer, all your problems will be nails.

If you are a frontend developer who only focus on frontend and you are brought up to a backend problem, you will only be able to provide the solution from frontend perspetive, same goes to vice versa. It tells me don't expect an overall best solution. To better equip ourselves, we should at least learn some perspectives from various ends, be a T-shaped person.

Not Invented-Here (NIH) Syndrome

NIH syndrome causes you to fall in love with your own ideas. Also, we tend to avoid things that we didn't create ourselves. This is true to certain extent that some of use might want to reinvent the wheel because we think that the things we do are far way better than the solutions provided by third party. I see this as a fallancy that we might easily face into, especially when we are building our own solution. When you face this situation, take a step back and then examine their quality and drawback.

Conclusion

It is hard to summarise this book in a sentence, because there are many cognitive biases and each of them act separately to different situations. If you expect to have steps or roadmap on how to think clearly from this book, well, you may look into another book. This book doesn't work that way. It gives you insights on your behaviour and action when you are facing any situations.

Surely there are cognitive biases that catch into your eyes too and feel free to share with me in the comments.

Happy coding. 💻

Delete Multiple Versioning-Enabled S3 Buckets by boto3

Andy Lim — Thu, 01 Apr 2021 04:32:16 +0000

Have you ever tried to delete multiple non-empty S3 buckets with versioning-enabled on AWS Console? How irritating it is to go through each of every S3 bucket? To delete the bucket on AWS console, you have to:

delete every objects in the bucket
empty the bucket by typing the bucket name
deleting the bucket by typing 'delete'

You have to go through 2 confirmations. Let's say you have up to 20 buckets to delete, you have to go through total up to 40 confirmations to make by going through each of every bucket, which is totally a hell. And hence, I choose to do it programmatically. I will walk you through this post to write a simple script to delete multiple buckets.

Prerequisite

AWS CLI installed. Download from here.
Configure AWS credentialds in AWS CLI. Can refer this document to learn how to configure your AWS CLI. Make sure you have the access to list buckets, delete versioned objects in the buckets and buckets.
Have your Git installed and enabled on your local machine.
pip installed. pip is a package manager for Python. Just like npm for NodeJS. Refer to this document to install pip.

Note that in this tutorial, I am using my administration account. It is always recommended to provide the least privilege permissions to the user.

Walkthrough

In this project, we will be using:

boto3: AWS SDK in Python
inquirer: To ease the process of asking end user questions, parsing, validating answers, managing hierarchical prompts and providing error feedback

Clone the repository and go to s3 directory in terminal.

git clone https://github.com/andylim0221/aws_scripts.git

cd aws_scripts/s3

Install the necessary packages such as boto3 and inquirer.

pip install -r requirements.txt

Run the script with your profile or default profile.

python delete_bucket.py # default profile

# or with profile any
python delete_bucket.py --profile any

And you will see the similar lists as below :

You can choose the buckets you want to delete by pressing space bar and navigating by up arrow and down arrow button. The selected options will be shown in yellow font color. By clicking space bar again on the selected buckets will remove it from the options.

Once you have finished selecting, press Enter button and go to next step.

You will see the question to confirm if you want to delete the buckets you chose earlier. By typing delete, the action is confirmed and it will proceed to delete the buckets.

And now you just have to wait for the deletion to happen and in the meanwhile, have a good cup of coffee.

When the script is done, check your buckets by listing your buckets to see if the selected buckets are deleted.

aws s3 ls #list all buckets

Note that, if you want to cancel the action or reselect the buckets, you have to kill the script by Ctrl-C.

Troubleshooting

When you list the buckets, sometimes the deleted buckets are still there, the reason why it is still in the list because the list is not updated immediately. You can check in your AWS Console and you will see the deleted buckets are shown as below :

Give it up to 24 hours, the bucket should just disappear from the list.

Conclusion

In this one simple script, I can :

remove objects
remove versioned objects
empty objects
delete buckets

Of course this can be improved in security or efficiency, but it should be doing well for the job. Let me know in the comments if you found bugs or issues from this solution.

Happy coding! 💻

AWS CloudFormation and Github Action

Andy Lim — Sun, 14 Mar 2021 08:09:54 +0000

Overview

The focus of this article is to introduce the integration of AWS CloudFormation and Github Action.

Introduction

Github Action is a CI/CD service provided by Github and it has free tier. With Github Action, you can choose to run CI/CD actions on your self-hosted server or Github docker. It's user friendly and there are multiple plugins available in Github Action Marketplace which reduce your time of development and avoid RTW (Reinvent the Wheel).

AWS CloudFormation is a service that gives developers an easy way to automate the infrastructure provisioning using Infrastructure as Code (IaC). What's so special about Infrastrucure as Code is that we are able to run tests because these infrastructure components are written in code.

In this demo, we will be integrating AWS CloudFormation with Github Action and do the following:

Any branch other than main:
1. Linting: Check the CloudFormation linting format using cfn-lint
2. Security check: Check the security of each components in CloudFormation using cfn-nag to validate if they are adhered to security best practices from AWS.
Main branch (protected):
1. Deploy: Deploy the CloudFormation template to S3.

Prerequisite

You may need the following to run on your projects:

AWS credentials with programmatic access, so you need your AWS Access Key, AWS Secret Key
AWS S3 bucket with bucket policy such that a put action is allowed and ACL is allowed, you need the AWS S3 bucket name and make sure it is globally unique.
Github account

Project Structure

.
├── .github
│   └── workflows
│       ├── master.yml
│       └── test.yml
└── cloudformation
    ├── conformance-pack.yaml
    └── example.yaml

For this project demo, I write two CloudFormation templates:

conformance-pack.yaml to deploy a set of config rules and remediation actions.
example.yaml to deploy a simple S3 bucket with SNS trigger

Nevertheless, the main focus is on the Github Actions. Let's take a look on what does the Github Action do.

Github Actions

As you can see from the project structure, there are mainly two templates, master.yaml and test.yaml under the .github/workflow directory. GitHub Actions uses YAML syntax to define the events, jobs, and steps and these files are stored in a directory called .github/workflows. In order for you to run the Github Actions, you need to create this directory so that whenever you push a commit into your repository, Github Actions will run automatically.

In this directory, create two files, master.yaml and test.yaml.

# test.yaml

name: 'Testing Github Actions'


on: [push]

jobs:
  Cloudformation-checker:
    name: Check linting and security concerns
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: cfn-lint-action
        uses: ScottBrenner/cfn-lint-action@1.6.1
        with:
          args: "cloudformation/**/*.yaml"

      - name: cfn-nag-action
        uses: minchao/cfn-nag-action@v0.1
        with:
          args: '--input-path cloudformation/

This Github Action will be triggered if a commit is pushed into this repository for all the branches. It will validate the template syntax using cfn-lint-action and run security check using cfn-nag-action if the commit passed the cfn-lint-action. What's good abou this Github Action is that all of these steps are ready-made plugin from Github Action Marketplace and they have been created and used by the Github communities. For this project, I'm using the plugins from ScottBrenner/cfn-lint-action@1.6.1 for cfn-lint-action and
minchao/cfn-nag-action@v0.1 for cfn-nag-action. There are several similar plugins in the Marketplace, please makes sure you read their README.md in order to enable the actions.

name: S3Deploy

on: 
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: "Checkout"
        uses: actions/checkout@v2
      - name: S3 Sync
        uses: jakejarvis/s3-sync-action@v0.5.1
        with:
          args: --acl bucket-owner-read --follow-symlinks
        env:
          AWS_ACCESS_KEY_ID: ${{secrets.AWS_ACCESS_KEY}}
          AWS_SECRET_ACCESS_KEY: ${{secrets.AWS_SECRET_KEY}}
          AWS_S3_BUCKET: ${{secrets.AWS_BUCKET}}
          SOURCE_DIR: './cloudformation'

Lastly, this Github Action will copy the files in the cloudformation directory to S3 bucket if a commit or a pull request is made into the main branch. As you can see from the env section, there are several secrets, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_S3_BUCKET. Please refer to this documentation site to learn how to input secrets into your Github Actions.

What's good about this is that you don't have to manually push the templates into S3 bucket by running multiple command line or browsing into the AWS management console.

Conclusion

Throughout this simple project, you can learn how to run a simple CI/CD pipeline to run simple template syntax check and security check on the CloudFormation templates and then push them into a dedicated S3 bucket. Of course, there are some improvements, for example, auto provision the infrastructures into different environments after a file is pushed to S3 bucket. Feel free to extend from here and hope to see more on CI/CD for Infrastructure as Code.

DEV Community: Andy Lim

UFW and Docker Security on Linux machine

Install UFW

Install Docker

Problem arises

What's the problem and how to fix it

Conclusion

The Art of Thinking Clearly - Book Review

Introduction

What's so special about this book

How does it relate to my career journey

Swimmer's body illusion

Déformation professionnelle

Not Invented-Here (NIH) Syndrome

Conclusion

Delete Multiple Versioning-Enabled S3 Buckets by boto3

Prerequisite

Walkthrough

Troubleshooting

Conclusion

AWS CloudFormation and Github Action

Overview

Introduction

Prerequisite

Project Structure

Github Actions

Conclusion

Reference