DEV Community: ant Kenworthy

Linux Firewall: Blocking a lot with a little

ant Kenworthy — Fri, 30 Aug 2024 17:00:00 +0000

I have a need to block a large list of ever changing IP addresses from servers and systems I operate.

The list of IP's I'm using can change hour by hour based on certain signals obtained from different logging and monitoring systems that are in place and some times the list of IP's can belong to an entire country!

Basic Blocking

I can block single things easily with iptables by using this on the command line:

iptables -A INPUT -s 1.1.1.1 -j REJECT

This will REJECT incoming connections from 1.1.1.1 ( Cloudflaire's public DNS resolver )

You probably don't want to do this as its just an example of a simple block command but the IP address listed there can be changed for something different or it can be changed for a network prefix for example 10.0.0.0/16

Grouping things

I can create an additional Chain in iptables to group a bunch of things together.

for example, I may want to filter things heading to my mail services
I'd create a new chain with the command:

iptables -N mail-services

Then I would point all of my mail services in that direction:

iptables -A INPUT -p tcp -m multiport --dports 25,465,587,143,993,110,995 -j mail-services

As before I can then add in specific rules for servers to the mail-services chain:

iptables -A mail-services -s 1.1.1.1 -j REJECT
iptables -A mail-services -s 8.8.8.8 -j ACCEPT

In this example I am rejecting connections from 1.1.1.1 but ACCEPTing connections from 8.8.8.8

Extending further

Now I have a chain setup for my mail services I can now start blocking ( or allowing ) certain sources attempting to connect.

I have a handy selection of IP addresses gathered from my monitoring systems of IP addresses that are stored safely within a database which I can access by running a local script.

The script returns a list of IP addresses from the database on each line.

The output of the script would look something like this:

1.1.1.1
1.1.2.2
192.168.1.0/24

etc

I make use of this script output by wrapping it in another shell script like so:

#!/bin/bash
## Flush the mail-services chain
iptables -F mailservices
for ipaddr in $(./spot-block-ips); do iptables -A mail-services -s ${ipaddr} -j REJECT; done

This will add in all IP addresses flagged to be blocked to the servers firewall after all the old ones have been cleaned out ("flushed")

CPU Overhead

This works really well when there's a nice handful of IP's to block from the mail services.

However, now the list has grown quite large there is a slight overhead on CPU time to do this now.

IPset's

Turns out there's a better way of doing firewalling with much bigger lists and that's by using ipset

From the man page for the command:

ipset is used to set up, maintain and inspect so called IP sets in the Linux kernel. Depending on the type of the set, an IP set may store IP(v4/v6) addresses, (TCP/UDP) port numbers, IP and MAC address pairs, IP address and port number pairs, etc. See the set type definitions below.

You may need to install that tool to use it. On Debian based distributions that will be apt install ipset

To simplify this process a little; if an IP address has broken one of our rules we'll now block it outright from all services rather than just the services it has attacked.

Create a new set

Run the following command to create a new ipset called blocklist:

ipset create blocklist hash:net counters

Note:
Its best to try and plan ahead here when you create a new set.
hash:net will define how the IP addresses are stored in the set inside the kernel

by specifying ip here, in place of net, it will allow for single IP's and network blocks of IP addresses.
be aware however; adding a netblock of 192.168.1.0/24 will add in 192.168.1.0, 192.168.1.1, 192.168.1.2 etc individually.
You may opt to select ip here in place of net depending on your needs.

The counters option here allows us to keep an eye on the number of hits we've had against an IP address in a similar fashion to how we would see this in iptables list entries ( iptables -L INPUT -v -n )

If items are present in the list but are getting no hits then the IP address could perhaps be flagged for removal from the block list.

Populating the list

We can very easily adapt the wrapper script from earlier to use this new set.

here is an updated version:

#!/bin/bash
for ipaddr in $(./spot-block-ips); do ipset add -exist blocklist ${ipaddr}; done

Depending on the number of IP address you have to hand this may take a moment to complete.

Using the list

Now the list is populated we have to make use of it.
As mentioned earlier we're not paying attention to the service the IP is trying to access anymore, we're just blocking all services for that IP instead.

Run the following to enable the ipset list:

iptables -A INPUT -m set --match-set blocklist src -j REJECT

Done !

The list is now in use on your system and you should be blocking the things you've decided to block! 🎉

Depending on the frequency of a connection attempt you should be able to see any new hits by running:

ipset list blocklist |less

Updating the list

From time to time you may want to add or remove items to the list

you can use the ipset add blocklist <ip> and ipset del blocklist <ip> to add and remove single items on the list.

You can also clean out the list with the flush option

The wrapper script in use here will add IP addresses to the list regardless of if they are present or not but will not remove them.

A fix for this would be to use a temporary list and then switch the lists round.

Example:

#!/bin/bash

## End the script if there's an issue
set -e

## Create a new temp blocklist
ipset create blocklist-tmp

## Add IP addresses to it
for ipaddr in $(./spot-block-ips); do ipset add -exist blocklist-tmp ${ipaddr}; done

## Swap the tmp list in place of the old list 
ipset swap blocklist-tmp blocklist

## Destroy the now old list
ipset destroy blocklist-tmp

This is a better way of performing updates to our lists as there is now no gap between the iptables flush command and the new IP addresses making an appearance.

Dedicated firewalling

You'll correctly ask why dedicated firewalls, Hardware or otherwise, are not in use for this.

While they are already in use for more static rules some service providers some times struggle with the number of entries and frequency of updates to the IP addresses in the system that are put through.

At time of writing there's about 7741 IP entries in the database with would mean multiple entries with cloud providers or many hours of work with others.

Summary

By switching the list of blocked IP addresses out of a multi rule iptables chain in to a hash list stored in the kernel we saved about 1% cpu overhead on the systems its deployed on.

🎉

Packer: Using an image family from another project

ant Kenworthy — Mon, 22 Mar 2021 21:45:23 +0000

Following on from my previous post: Packer: Building images on Google Cloud You should now be able to build compute images on GCP and you may have used this to build a pre-configured image you can effortlessly deploy again and again without having to wait to install software.

You may now want to use that image, or someone else's image from a different project as a starting point for your web or database server.

If you do you, can add the following line to your packer template in the builders section to specify another project to source your image from:

source_image_project_id

Here's how that may look when used in your packer template:

{
  "builders": [
    {
      "type": "googlecompute",
      "project_id": "my project",
      "zone": "us-central1-a",
      "source_image_family": "base-image-linux",
      "source_image_project_id":, "example-project",
      "ssh_username": "packer",
      "image_name": "packer-{{timestamp}}",
      "image_family": "socks-web"
    }
  ]
}

Here the base-image-linux may contain our companies standard logging setup and common utilities. From this you could then install your web server or database utilities on top of that common image.

Another example may be to use the public images provided by a regulatory body where you need to add some helper scripts before you deploy it to your project.

🎉

Google Cloud: IAM Conditions

ant Kenworthy — Mon, 15 Mar 2021 23:32:55 +0000

We can use IAM to control who has access to what within our project and who can do what to things like storage buckets, but what if we wanted to restrict when someone could do something or to what object in a bucket. That's where IAM Conditions come in to play.

What are IAM Conditions ?

IAM conditions are one or more rules written in "CEL" or Common Expression Language. Each rule must evaluate as true before the defined access role associated with it is permitted.
These rules can specify the type of resource we want to control, time, date's and some string operations.

Here is a simple example:

resource.type == "compute.googleapis.com/Instance"

Here we are expecting the resource type we're trying to use to be an instance ( which are part of the compute API ). If the resource we are trying to use is something else this condition will fail and our request will be denied.

There's more information about CEL and its specifications here

Can you give some examples ?

Sure, here's one which will restrict access to a specific time period and between certain days:

request.time.getHours("Europe/London") >= 9 &&
request.time.getHours("Europe/London") <= 17 &&
// Days of the week range from 0 to 6, where 0 == Sunday and 6 == Saturday.
request.time.getDayOfWeek("Europe/London") >= 1 &&
request.time.getDayOfWeek("Europe/London") <= 5

When combined with roles/compute.admin it would restrict the user and/or group to only be granted the role during office hours ( Monday to Friday between 9am and 5pm ).

Here's another that can be used to restrict what objects in a storage bucket can be manipulated:

resource.type == "storage.googleapis.com/Bucket" &&
resource.name.startsWith("projects/_/buckets/inputbucket-001/objects/example-folder")

When combined with the role roles/storage.objectCreator it would restrict the user and/or group so that its only able to create new objects in our bucket named inputbucket-001 in the folder example-folder. Uploading to the root of the bucket or any other root level folder would be denied.

Are there any limitations to using IAM conditions ?

Yes, there are some:

They cannot be used with basic roles ( formally "primitive" ), allUsers or allAuthenticatedUsers
They only work with certain services at the moment. Check the documentation here for more information.
There is a maximum number of 12 logical operators in one condition expression
There is a maximum of 20 role bindings for same role and same member
There is a maximum of 100 conditional role bindings

Attempting to exceed these limits will return a fail message when you apply your change.

Also, conditional role bindings will not override role bindings with no conditions. If a user and/or group is bound to a role which does not have a condition, then the member always has that role. Adding the member to a conditional binding for the same role will have no effect

Can this be applied via terraform ?

Yes! Here's an example directly from the terraform documentation:

resource "google_project_iam_member" "project" {
  project = "your-project-id"
  role    = "roles/firebase.admin"
  member  = "user:jane@example.com"

  condition {
    title       = "expires_after_2019_12_31"
    description = "Expiring at midnight of 2019-12-31"
    expression  = "request.time < timestamp(\"2020-01-01T00:00:00Z\")"
  }
}

As you can see from the description field any requests made after 2019-12-31 by jane@example.com will be denied.

🎉

Terraform: Getting started with GCP

ant Kenworthy — Fri, 12 Mar 2021 22:34:35 +0000

What's Terraform?

Terraform is a tool created by Hashicorp which allows us to describe our infrastructure in easily readable code. This code can also be stored in our source code management system to allow us to track changes to our infrastructure over time.

Terraform works by talking to the API endpoints Google provide for the platform to setup our desired infrastructure.

Where do I start ?

First we'll need to configure a "provider" so we can talk to the platform.

You'll need to put the following in your terraform file:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~>3.57"
    }
   }
}

provider "google" {
  project = "my_Project"
}

In the terraform block above the first thing we're doing is setting up our required providers, where to get the google provider from and the version of that provider.

When we specify the source as hashicorp/google terraform knows to get the provider from the terraform registry.

While setting the version options isn't required, it does help us to track any changes and mitigate possible breaking changes when applying our infrastructure.

Above we're setting the version of the provider to 3.57 however we are also allowing for any minor bug fixes by using the ~> prefix in the version field.

More info on version constraints in terraform can be found here

In our provider configuration we're also setting our default project to my_Project.
Setting the default project here will prevent us from having to set it for each google resource we define.

How will Google know who I am?

As with any good cloud provider Google won't just let any one through the door. So we'll have to let Google know who we are first before we can do anything else.

When we're using terraform there are couple of ways to do this:

Application Default Credentials (ADC)

With the Google Cloud SDK installed on your system; You can run the following commands on your computer to login to google cloud and set your default application credentials:

First you'll need to login: gcloud auth login

If you're on Windows, Mac or Linux (FreeDesktop.org compatible) it will open your default web browser and prompt you to login to google cloud.

Once complete, you'll need to run gcloud auth application-default login to trigger
a similar process and set your default application credentials.

If a web browser cannot be opened automatically you will be presented with a URL you can copy and paste into your browser.

If you have multiple browser windows logged in to different accounts it may be a good idea to add the --no-launch-browser command line option to prevent gcloud from trying to open the URL for you. You can then copy and paste this in to the browser session you'd like to use for your login.

Credentials file

A Credentials file contains a private key we can use to talk to Google’s API’s and is linked with a service account that has been allocated specific set of permissions.

More information about creating and downloading the service account key can be found here

We can reference our credentials file by adding the following line to our google provider configuration:

Provider “google” {
  credentials = “/path/to/keyfile.json”
  Project     = “my_Project”
}

Alternatively we can also reference our credentials file by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable, for example:

export GOOGLE_APPLICATION_CREDENTIALS=”/path/to/keyfile.json”

Where you need to make use of a service account key; using the environment variable would be a better option so we can avoid having to change the path to the key file for each person or system that uses it.

It's recommended that a user account is used outside of any automated pipeline operations as these are linked to people and can be deactivated when a user leaves a project or if you need to consult the logs to see who created a resource.

Now you're setup you can start making use of the platform's resources and benefits!

🎉

Packer: Building images on Google Cloud

ant Kenworthy — Tue, 08 Sep 2020 00:29:57 +0000

Installing the software we need to run our website or tools, can take some time to complete. This isn't ideal when our deployments are time sensitive, like when we need to scale up the web server pool to handle that sudden interest in your latest line of fancy socks.

We can solve this issue by pre-installing the software we need and setting the configuration options, which we can then turn in to an image we can deploy over and over again.

To make this entire process quicker, repeatable and something we can easily template - We'll use packer

What is Packer?

Packer is a Hashicorp tool that can be used to ease the creation of system images on cloud platforms or for other local tools like Vagrant ( another Hashicorp tool), Docker and VirtualBox.

Where can I get it ?

You can download the HashiCorp Packer tool from the download page on the packer website

There's also a handy guide on the hashicorp learning portal for how to install it on your operating system

How can I use it with the Google Cloud Platform?

You'll need to create a "builder" configuration in your packer file.
Here's a basic example:

{
  "builders": [
    {
      "type": "googlecompute",
      "project_id": "my project",
      "zone": "us-central1-a",
      "source_image_family": "debian-10",
      "ssh_username": "packer",
      "image_name": "packer-{{timestamp}}",
      "image_family": "socks-web"
    }
  ]
}

The options we have set above do the following things:

type
is the type of builder we're using. Here its googlecompute because we're talking to GCP

project_id
is the project where the final image will be stored

zone
Will be the zone we build our image in

source_image_family
This is the group of images where our base image will be taken from

ssh_username
The ssh username we'll use to talk to the instance that will create our image

image_name
The name of the image we will create.
The image name needs to be unique within our project so we're using the packer variable timestamp to do that.

image_family
The name of our image family. This is optional but is useful later on.

What is this `image_family` thing ?

Images can be stored in groups referred to as image families. When building instances we can ask GCP to get our image from our family which will return the latest image that was added to it.

If something is found to be wrong with a new image we can tag it as deprecated and the family will offer up a previous version next time we deploy an instance.

More info on image families can be found in the GCP documentation

How do I install that web server in my image?

You'll need to add a provisioner or two, to your packer config file.

Provisioner's are usually shell scripts or file uploads but could be a puppet, ansible, salt or chef operation.

Here's a complete example of our packer file including our new provisioners:

{
  "builders": [
    {
      "type": "googlecompute",
      "project_id": "my project",
      "zone": "us-central1-a",
      "source_image_family": "debian-10",
      "ssh_username": "packer",
      "image_name": "packer-{{timestamp}}",
      "image_family": "socks-web"
    }
  ],
  "provisioners": [                                                           
      {                                                                         
      "type": "file",                                                           
      "source": "./index.html",                                           
      "destination": "/tmp/index.html"                                            
      },
      {                                                                         
        "type": "shell",                                                        
        "inline": [                                                             
            "sudo apt-get update -qq",
            "sudo apt-get upgrade -qq",                                     
            "sudo apt-get install -qq apache2",
            "sudo cp /tmp/index.html /var/www/html/index.html",
         ]                                                                      
    }]
}

Here we copy our index.html file to /tmp, Update the cache for apt, perform a system upgrade to make sure we have all the updates installed then install apache and finally copy our index page in to place.

How do I build that image ?

You can now run the following command from the directory where you've saved the packer file:

packer build example.json

Packer will now go away and build that image for you and will store the resulting image as packer-xxxxxxxxx ( where xxxxxxxxx is a timestamped value ) The resulting image will be linked to the image family socks-web which you will then be able to reference in your terraform template for your project.

🎉 Tada!

Are there other things to know ?

Yes, there are a couple of things:

Networking

The example above assumes the project you are using contains a default networking setup with subnetworks available in your chosen zone. If these things aren't available the build process will fail and you'll need to add a network OR subnetwork option to the builder config which lines up with your chosen zone.

It also assumes that the default firewall rules are in place so packer can ssh to the instance it creates.
If you don't have the needed firewall rules to allow access your build process will time out while waiting to connect.

`machine_type` has been omitted from the example

It will default to n1-standard-1 but you may need to increase its size depending on what you're doing. A larger machine type will help it run faster.

Cost

There is a cost for storing your images.
At time of writing its about $0.050 for US and EU multi region storage but it can be as high as $0.075 is specific regions.

There will also be a cost for building the image.
This will vary based on the image size you use and where you choose to build it.
At time of writing the cost of the the given example should be in the region of $0.04749975 Per hour

Check the pricing page for more info.

To quote the pricing pages:

If you pay in a currency other than USD, the prices listed in your currency on Cloud Platform SKUs apply.

❗ Make sure you check the cost before you run up a big bill ❗

Cleaning up

If you don't want that image you've just created you can head over to images in the compute section of the console and delete the image you no longer want.

Note: You cannot delete public images from your project - They are free anyway 😃

SSH: Multiplexing

ant Kenworthy — Sat, 11 Jan 2020 22:37:22 +0000

What is it ?

Multiplexing, sometimes referred to as muxing, is a way of making use of one connection for a number of signals.
The internet connection you're reading this on may have been multiplexed at some stage along its physical route to your building or local communications tower and can used to reduce the overall cost of having to lay multiple cables to endpoints.

There is an article on Wikipedia about multiplexing which goes in to more depth about what and where else in the world its used.

What is the link to SSH ?

We can use multiplexing for our SSH connections to help speed up processes where multiple commands need to be executed on the same server or via the same jump host.

How does SSH do this ?

Your SSH client can create a control ( 'master' ) connection to the server which can then be used for multiple sessions without having to create a new TCP connection for each one. OpenSSH shares this connection via a socket that it can create on disk.

How can I do it ?

You can either do this via your command line by setting some options at run time or you can add some configuration options to your SSH client configuration file.

For ease and and some clarity; here's a quick configuration example for our jump-box:


Host jump-box
  HostName jump-box.example.org
  ControlPath ~/.ssh/controlmasters/%r@%h:%p
  ControlMaster auto
  ControlPersist 10m

Once this is in place the next connection you make to jump-box will create a control connection and then create a session within in it which will contain your terminal.

The important configuration bits here are the ControlPath ControlMaster and ControlPersist options.

`ControlPath`

Sets the place the socket file will be created. openSSH uses this to talk to the control process. The %r %h and %p tokens correspond to the remote username, remote host and the port of the remote host respectively. As per our example this will create the control socket at ~/.ssh/controlmasters/user@jump-box.example.org:22.
This ensures you get a different socket for each connection you make.
See man 5 ssh_config for more info on the different tokens you can use here.

`ControlMaster`

Sets how SSH will use the multiplexing. Here we set it to auto so that our client will first check for existing connection and make use of it if one is found and if none are present one will be created. see man 5 ssh_config for info on other options.

`ControlPersist`

This tells SSH how long it should keep a control session open for after the last session has closed. We're using 10m in case we've forgotten something or want to repeat a task.

How can I use this for all of my SSH connections ?

You'll need to adjust the configuration you put in to your SSH configuration file like so:

Host *
  ControlPath ~/.ssh/controlmasters/%r@%h:%p
  ControlMaster auto
  ControlPersist 10m

All of your new SSH connections will now be multiplexed

Controlling Port forwarding when multiplexing

Port forwarding and related controls are transferred to the control session. So restarting a SSH session with the same port forwarding options set will show an error message about the port already being in use.

The same controls you would expect to be accessible via the escape sequence are not present either.

Instead you have to control these features via the -O command line option. Here are some examples:

To add a new forward without starting a new session:

ssh -O forward -L 8080:localhost:80 jump-box.example.org

Removing a port forward:

ssh -O cancel -L 8080:localhost:80 jump-box.example.org

Note: If you specified a user on your initial connection ( user@server ) you also have to specify it here to find the control socket

Unfortunately there doesn't appear to be a way to list the existing forwarding options you may have set.

Server side support

The only service I've been unable to use multiplexing with is bitbucket and the reasons for why are noted here

You can remove hosts from the multiplex configuration by adjusting the Host line, for example:

Host * !bitbucket.org

This would remove the bitbucket domain from this host configuration block

Happy Multiplexing ! .o/

SSH: Tunneling to remote servers

ant Kenworthy — Tue, 24 Dec 2019 16:59:32 +0000

When creating our infrastructure, either in the cloud or on prem, we ~~often~~ usually need a way to connect to machines running behind the scenes to do things, such as upgrades or fixing problems.

Typically the tool of choice for most remote system administration work is ssh. openssh is probably the most common version of ssh that is used on systems today and is often pre-built and installed on your operating system.

When we need to connect to our web servers that are safely tucked behind a firewall on local IP addresses ( RFC1918 ) we can use SSH to tunnel through our bastion or jump-host, here's how:

Here's a basic diagram of how I'll be connecting to a web server:

  { Internet } ------ [ Jump-box ] ------ [ Web-server-1 ]

I need to connect to web-server-1 to fix an issue but it doesn't have direct exposure to the internet. I can use the jump-box to make that connection like so:


ssh -J jump-box.example.com web-server-1

That's it. It really is that simple!

You may notice that the webserver isn't using a fully qualified domain name such as web-server-1.example.com. This is because the ssh client will make use of the DNS configuration of the jump host to find web-server-1. If the DNS isn't configured properly on the jump-box, or you make a typo, you'll get an error message back saying it couldn't be found.

You can add as many servers to the -J option as needed in a comma separated format. The highest number of jumps I've made is 3. The client will connect to each of the servers in order to establish a connection to the end point web-server-1. An example maybe:


ssh -J jump-box.example.com,jump-box-2 web-server-1

Note : The -J option was introduced in openssh version 7.3. If you have an earlier version of the client the command line option may not be available to you.

The -J option is a shortcut for the ProxyJump configuration option. That you could find in your ssh client configuration files already ( .ssh/config for user configurations or /etc/ssh/ssh_config for system wide )

Example:
In my ssh client configuration it would look like this:


host web-server-1
 ProxyJump jump-box.example.com

This would create an alias for web-server-1 that the ssh client can use. Now I'd only have to type ssh web-server-1 and ssh would take care of the rest.

But wait, there's more!

An alternative method for doing this would be to use the proxy-command option:


ssh -o ProxyCommand="ssh -W %h:%p jump-box.example.com" web-server-1

This first starts an ssh connection to the jump box and then, with the -W option, requests that standard input and output on the client be forwarded to host on port over the secure channel

Sometimes the ssh client you have may not have these features or they could be restricted in some fashion. For this we have to revert to good old fashioned port forwarding.

In our first terminal we would run this command to setup a connection to the jump box but at the same time we ask it to listen on port 2222 on our local machine and pass those connections through to port 22 on web-server-1:


ssh -L 2222:web-server-1:22 jump-box.example.com

Then in our second terminal we would run this to initiate a connection to port 2222 on our local machine:


ssh -P 2222 localhost

Using port forwards in this way works well but becomes cumbersome if you need to connect to more than one jump-box or more than one endpoint within your network.

Ta-da! \o/

tldr: use ssh -J jump-box.example.com web-server-1 to tunnel ssh connections to web-server-1 via jump-box.example.com