I. Introduction
Would you like to hear the good news or the bad news first?
- Let's begin with the downside. If you've solely relied on securing the internal cluster network with NetworkPolicy during your recent on-premise K8S installation, you've missed out on 50% of the security measures. Many threats originate from within, but they could just as easily infiltrate from outside. Bummer… I know!
Hey! At least you did half of it! I'm sure that some people reading this blog are like "Huh what are NetworkPolicies anyway ?". Those cannot be saved… May god have mercy on their souls.
- Now to the good news part! Securing incoming requests to your cluster can also be achieved and I've got you covered!
Why you should shield your cluster from the exterior world?
As mentioned previously, numerous threats may arise from within. For instance, the act of pulling and executing a rogue Docker image could potentially corrupt your nodes, propagate across applications, and ultimately compromise your entire infrastructure.
⬆️A container that seems fishy; Something doesn't look right…🤨
Let's not dwell on that frightening idea any longer and address our specific concerns!
Kubernetes is composed of masters, workers and ETCD nodes. All these components are working from inside virtual machines (VMs) that are part of a network.
⬆️Network around Kubernetes components
The network that connects these VMs has 3 purposes:
- Allow each VM (K8S component) to communicate with each other (In green 📗)
- Allow developers to interact with Kubernetes to deploy their payloads (In blue 📘)
- Allow engineers to connect to the underlying VMs that support K8S for debugging and stuff (In orange📙)
Securing this network is of paramount importance as one of the VMs could get infected and spread to all others.
On one hand, if your VMs are part of a private network then you might feel safe but in reality, it's private until it's not… And that's when sh*t hits the fan!
On the other hand, if your Kubernetes cluster is available directly from the internet then you really should read this tutorial carefully as all your nodes might currently be screaming H4CK ME PLEASE!
⬆️Hackers attacking exposed VMs that have no protection from incoming requests
There are only two ways to access a system. Physically or from a network endpoint. So unless you find yourself tied up to your chair with a man equipped with a malevolent USB key, the primary threat will definitely come from the network.
⬆️If that's you then you and your clusters are in deep trouble!
In general, hackers search for open ports on the exposed VMs. As most ports are dedicated to a specific program or protocol, for instance, port 22 is always SSH, listing opened ports gives a good idea of what runs on the target. With this list, the hacker can then search for an exploit and hack his way in. So the fewer open ports, the better.
Benefits of limiting opened ports on underlying VMs:
- protects your infrastructure from infection
- slows propagation of malware in case of infection
- allows better control of what (custom) software runs on the VMs
What technology to use for restricting incoming requests and logging them
On Linux, there are many ways to secure incoming network requests. It's called "firewalls". You may have heard of "firewallD" or "UFW" (Uncomplicated Fire Wall (it's a lie)). The truth is, I can't just give you the instructions for all existing firewalls. Moreover, I did tests using a few and didn't quite achieve adequate results and broke the Kubernetes cluster quite a few times.
In the end, I think that the best technology to use when it comes to networks and you don't have one hundred rules to write is: "iptables"!
I'm sure you've heard of it. It's older than you. It was created during the Jurassic era but is still relevant today.
⬆️iptables hype during Jurassic
Even better, most firewalls are simple overlays of iptables, manipulating their rules for you like a wand and dark magic.
But fear not. Old doesn't (always) mean that it sucks. I have mastered the beast and if you read thoroughly the next sections I'm sure that I can pass on this knowledge to you as it was passed to me by an old but wise sage.
II. How to use iptables dark magic?
A network metaphor that you'll love
When a happy network packet is received by a VM it goes through the iptables administration. The packet will go through about 7 different rooms before reaching its destination.
⬆️The process of a network packet
Each room above is called a "Table" in iptable jargon
Each room is responsible for evaluating the network packet and decides if it can continue the process onto the next room or gets kicked out!
To achieve this selection each room has multiple desks to which the packet must report. Each room has about 1 to 5 desks depending on the room type.
Those desks are called "chains" in iptable jargon
Sat to those desks, are overly suspicious old ladies analyzing packets. They have a list of rules that a packet criteria can match and take action if it does.
⬆️Meet "Ms. Input" with her suspicious look. No weird packets at her desk will go unnoticed by this expert eye…
If the packet matches one of the criteria (rules) then two things can happen:
- It can be ACCEPTED immediately and go to the next room even if there are other desks left in the room. Its process can continue! Wishing it good luck with the next room!
- It can be REJECTED and gets killed on the spot! 😱
IMPORTANT CONCEPT HERE
=> By default, if the packet doesn't match any rules at any desks, it will still arrive at destination! This behaviour is called "default accept". When we finish setting our rules to match the packets we trust , we will change the default behaviour to "default deny". This means that if a packet didn't match any rules at any desks it will be killed by default when leaving the last desk available!
But, good news traveler!
One positive aspect is that we're solely focused on one room and one desk! And that's fantastic - do you know why? Because I don't come close to understanding what half the other old ladies do and neither will you (or you wouldn't be reading this tutorial in the first place).
So let's focus on the only room and desk that interests us. It's the Filter room!
⬆️In red is the only room of iptables that we will cover
Understanding the filter table and INPUT chain
Let's lose the metaphor for a bit and repeat what is what. A room is called a "table" which the binary "iptables" gets its name from. And each desk is called a "chain". You'll get used to it.
For some reason, chains are always written in caps. I guess that the guy who invented iptables was so old and deaf that people constantly yelled at him so he could hear them. Hence the cap! It's just a theory though…
The filter table (room) is all about filtering... That you may have figured out and you're right, WP! That's perfect because we want to filter incoming packets!
The Filter table has 3 default chains (desks)
- INPUT
- FORWARD
- OUTPUT
⬆️We are only interested in Mrs. Input and her list of rules
And do you know what chain (desk) a paquet would go through first when entering the Filter table? The…. THE…… INNNNNPUUTTT chain. Yes, yes. Well done.
The chain concept is easy to grasp. It's like functions in programming but instead of containing actual code they contain rules. The INPUT, FORWARD and OUTPUT chains are the default ones but you can create your own chains! You can have your own desk and an old lady just for your use case! Isn't that great? You just have to call this new chain from the INPUT chain like a programming function would!
Checking our current configuration
Let's see how are our iptables are configured using the following command:
iptables -xvnL
⬆️The content of our "filter" table and 4 chains
At first, this might be a bit overwhelming with information but if you decompose it chain by chain it's quite easy. It's composed of:
- The 3 default chains of the filter table: INPUT, FORWARD and OUTPUT
- A custom chain called DOCKER added by the docker engine
We can see that the INPUT chain is empty. There are no rules so no old suspicious lady will look at our packets… ☹️
The FORWARD chain on the other hand has a lot of rules but OUTPUT has none. Then the custom DOCKER one has only 2.
ℹ️ Remember that the Filter table is displayed by default because it's the most commonly used. However, you can list other tables if you wished using "
-t <table_name>".
Order of evaluation of rules
That's a simple one! All incoming packets are evaluated by the rules from top to bottom.
⬆️For example, using the FORWARD chain
In the example above, we can look at the FORWARD chain that has 14 rules. Each packet will undergo evaluation against rules 1 through 14 unless it matches one of these rules before reaching the 14th (If so it will either be ACCEPTED or REJECTED!).
The Macdonald Default Policy
Macdo's commercials are always like "Come as you are". Well… By default, iptables is the same. If the incoming packet doesn't match any rules and attains the last rule available then it will continue its journey whether it's legit or not. So, by default, iptables is useless, L.O.L.
But, it's good news because we don't want to mess things up with a wrong rule. It's only when we feel we are ready that we'll change the default accept to a default deny. When this happens, all packets that were not explicitly accepted will be dropped (killed ☠️).
You can look for the current Policy displayed in the previous screenshot. Look at the first line and you'll see: chain INPUT (policy ACCEPT... The last step of this tutorial will be to change this to (policy DENY....
Anatomy of a rule
Let's take an incoming packet with the following attributes:
- Protocole: TCP
- Target port: 22
We receive a TCP packet on our VM. It's on our port 22. We can search what program listens on our port 22, if any, with the command:
$ netstat -tupln |grep 22
...
openssh * * 22 7
This shows that a process named OpenSSH listens on our port 22 with PID 7.
It's logical as port 22 is commonly used by SSH! Let's accept this packet explicitly and allow OpenSSH to deal with it.
To allow the packet we can use the iptables command line as follows:
iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
💁Do not execute this command yet
What does all of this mean 😨 ?
-
iptables: It's the binary, I hope you understand that 😁 -
-A INPUT: Means "append" this rule to the INPUT chain. You don't have to specify a specific table because "filter" is selected by default (convenient!). -
-p tcp: Means, filter packets on their protocol type. Can be tcp, udp, icmp, or all. If the incoming packet had been an UDP packet then this rule wouldn't match it and the packet would move on to the next rule. -
--dport 22: Means, filter on packets that have a destination port set to 22. If our packet was meant to be addressed to another port (80, 53, etc) then this rule wouldn't match it. -
-j ACCEPT: Means, what action to take when the rule matches a packet. Can be "accept", "drop" or jumping to another chain.
Preventing failures
🎉Good news! If you inadvertently lock yourself out by blocking SSH or else, you should know that by default all iptables are restored when the VM reboots! So if you get locked out, simply reboot using the hypervisor. All clouders and VM providers have a button somewhere to hard reboot a VM. No need to destroy it!
Also, you should save the current rules you have just in case :
iptables-save >> iptables_save.log
Adding a custom chain
We want to avoid putting our custom K8S rules in the INPUT chain. Instead, we'll create a new chain that will be dedicated to Kubernetes stuff and will be called from inside the INPUT chain.
Let's create a new chain in the filter table called KUBE-PROTECTION
iptables -N KUBE-PROTECTION
Now if we list our chains again
iptables -xvnL
⬆️Our new custom chain has been created
We can see the KUBE-PROTECTION chain but it's empty.
Adding rules to a custom chain
For example, let's say we want an "allow SSH" rule to be stored in the KUBE-PROTECTION chain. We would use the command like this:
iptables -A KUBE-PROTECTION -p tcp -m tcp --dport 22 -j ACCEPT
It's simply "-A "
Listing our rules:
iptables -xvnL
⬆️A rule has been added to our custom chain
Yahoo! The rule is correctly stored in the KUBE-PROTECTION chain.
It's fairly easy to read:
- pkts: 0 => Is the number of packets that have matched this rule. For now, no packets have matched.
- target: ACCEPT => Means that if the packet matches then it's accepted and can directly go through to the next iptable's table
- prot: TCP => The protocol the packet must match to be evaluated by this rule
- source: 0.0.0.0/0 => Means that the packet must be sent from a specific IP address so the rule can match it. "0.0.0.0/0" means all source IPs can match.
- "dpt:22" => Means that the packet must be sent to our port 22 to match this rule.
Calling our custom chain
Having a custom chain to store custom rules is great. But for now, our incoming packet will not magically be redirected to our custom chain. As I mentioned earlier chains act like functions in programming. So we must call our custom chain from the INPUT chain (which, remember, is always the first chain to be called in the filter table)
iptables -A INPUT -j KUBE-PROTECTION
This might look a bit weird.
- The "
-A INPUT" means: Add a rule to the INPUT chain. - The "
-J KUBE-PROTECTION" means: Let's take the action of jumping to a chain called KUBE-PROTECTION. This might feel strange because previously we saw that -j was meant to ACCEPT or REJECT a packet. Fortunately, this is the last action that can be specified with -j , there is no fourth option.
Now if we take a look at our filter table:
iptables -xvnL
⬆️We added an action rule in INPUT to jump to KUBE-PROTECTION
We now have 1 rule in our INPUT chain. Its "target" is "KUBE-PROTECTION". So any packet that reaches this line will jump to the specified chain.
We can also see that the number of matched packets ("pkts" circled in red) in our custom rule is now rising. As I'm using SSH, all my packets on port 22 are now matching this rule and being accepted!
In programming, functions automatically return to their caller when finished. It's the same here.
When all rules inside our KUBE-PROTECTION chain have been executed and the packet hasn't matched any rules then the execution goes back to the INPUT chain to the next rule.
Note that it could also reach another jump rule and continue to yet another custom chain…
Removing a rule
For many reasons, you may need to remove a rule. The simplest way is to list the rules with their number and remove the rule you want using the specific ID.
iptables -xvnL --line-numbers
Let's remove the "allow SSH" rule from the KUBE-PROTECTION chain. It's rule number 1.
iptables -D KUBE-PROTECTION 1
It's pretty simple: it's the chain name followed by the rule number to remove.
A few more useful iptables options
Before we dive into the specific Kubernetes use case let's see one last set of options:
iptables -A KUBE-PROTECTION
-s 10.11.12.13,10.21.22.23
-i eth0
-p TCP
-m multiport --dport 2379:2380
-m comment --comment "This is a comment"
-j ACCEPT
A few new selectors here:
-
-s <IP,IP,...>: Means, match only packets coming from one of these source IPs. If the packet is not sent from one of these IPs then the rule doesn't match and the packet carries on. -
-i <network_interface>: Means, match only packets that are received on this specific network interface. Depending on your OS and version you will have one typical default interface name. Either "eth0", "ens5" or "enp". Newer systems tend to use "ens5" and older "eth0". Note that your VM will probably have other network interfaces but we won't care about them. Other processes deal with them.
To find your main interface you can use :
ip -a
# or
ifconfig
These will list your network interfaces and you should find a familiar name lost in there (eth, ens, enp)…
-
-m multiport: The "-m" is tricky because it can be used multiple times in one command to activate further options which is kind of dumb. In this case, it makes the next option possible. -
--dport <portA:portB>: Means allow all ports in the range A to B. Port ranges are available due to the previous use of the "-m multiport" option. -
-m comment: Means, active the comment option… I think that you see why this whole "-m" stuff is silly. Why not just allow comments on all rules… Brrr. Though I won't complain to the dead guys who developed this in 1999. At least it works! -
--comment "<Some comment>": Self-explanatory. You should comment on all your rules to describe what protocol or software they cover. For instance, for the "allow 22" rule we should rewrite the rule with
-m comment --comment "Allow SSH connectivity"
III. Building our Kubernetes protection rules
Strict minimum rules to have in the INPUT chain
We'll add all the Kubernetes stuff in a private chain and call it from the INPUT chain. However, you should check that your INPUT chain has the following rules evaluated before jumping to your custom chain:
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -j ACCEPT -p icmp
iptables -A INPUT -j KUBE-PROTECTION
Before applying these rules check if you don't already have them using iptables -xvnL.
-
Line 1: (BORING STUFF, CONSIDERING MOVING ALONG) The
-m contrackmodule allows us to use the following option--ctstate. It has two values: "related" and "established". The established field allows currently opened connections to stay alive. The related field on the other hand is used when a server returns an answer (to a request) on a different port than the one it was received on. For instance, you could have an FTP server that receives a packet on port 21 and answers on port 1234 (strange… I know!). If you don't allow related connections then the server won't be able to answer! This is quite frequent so this rule is a must-have. - Line 2: Accept packets on the loopback interface which basically means that you can talk to yourself. If you're a human then maybe consider a psychotherapist but for machines, it's all good!
- Line 3: Accept SSH on port 22. Don't lose this one…
- Line 4: Accept packets of type ICMP. ICMP is the protocol used by ping to allow anyone to ping your VM. You should remove it if you're directly connected to the internet. But if your VM is part of a private network then it's better to allow ping. It's good to know if the VM is alive or dead and ping does an amazing job of that.
- Line 5: If a packet has not been accepted or rejected yet and attains this rule it jumps onto our custom chain KUBE-PROTECTION.
What are the known legit Kubernetes packets to allow?
Kubernetes provides information about ports on VMs that must be opened to allow it to work properly: here.
⬆️Needed ports for Kubernetes to work taken from the documentation
Based on the above screenshot, the allowed ports to listen on our 3 types of VMs depend on their role:
Master nodes:
- 6443 TCP => Main entry point for API-server
- 10250 TCP => Kubelet
- 8472 UDP (Canal/Flannel VXLAN overlay networking)
- 30000:32767 => Node ports services
Worker nodes:
- 10250 TCP => Kubelet
- 30000:32767 => Node ports services
- 53 TCP/UDP => CoreDNS
ETCDs:
- 2379:2380 TCP =>Main entry point for ETCD communication
Methodology to authorize Kubernetes-related requests and not break everything
All that is not Kubernetes-related network must be investigated!
I'm 99% sure that if you only open the above ports and switch to "default deny" then things will break…
That's why we must monitor incoming packets and understand what is sending them to us and why! For instance, your VMs could have monitoring software checking if the VM is healthy, and cutting off all accesses to these legitimate external services will go BOOM…
So we need a way to log these packets and decide if we write a rule to match them or not. If we don't explicitly reject them they will be rejected by the "default deny" when we activate it.
We can achieve this by adding a logging rule. Each packet that goes through this rule will be logged with all its details :
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6
This rule will be added at the end of our chain and then log packets that none of our rules matched.
⬆️A log rule has been added at the end of our chain; Also note that I have added to INPUT the rules mentioned earlier
Now let's look at the logs:
$ journalctl -f |grep iptables
Jan 29 23:02:06 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=143.188.50.238 DST=192.65.42.44 LEN=52 TOS=0x00 PREC=0x00 TTL=42 ID=53443 DF PROTO=TCP SPT=35163 DPT=22 WINDOW=1191 RES=0x00 ACK URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=137.114.37.1 DST=192.65.42.44 LEN=32 TOS=0x08 PREC=0x00 TTL=5 ID=1 DF PROTO=ICMP TYPE=8 CODE=0 ID=7479 SEQ=1
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=82.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24788 DF PROTO=TCP SPT=49834 DPT=1234 WINDOW=1026 RES=0x00 ACK PSH URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=76.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24789 DF PROTO=TCP SPT=49834 DPT=6443 WINDOW=1026 RES=0x00 ACK PSH URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=30.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24790 DF PROTO=TCP SPT=49834 DPT=21 WINDOW=1026 RES=0x00 ACK PSH URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=27.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24791 DF PROTO=TCP SPT=49834 DPT=22 WINDOW=1026 RES=0x00 ACK PSH URGP=0
(scroll to the right)
There you'll see all the packets that didn't match any of our rules. If we had activated the "default deny" they all would have been dropped (killed 😵).
Anatomy of a packet log
Taking one random line from above and removing useless information for convenience:
IN=eth0 SRC=82.242.156.152 DST=192.65.42.44 PROTO=TCP SPT=49834 DPT=1234
What you should extract from each log line is:
- SRC: That's the IP who contacted you.
- DST: It's… You. So we don't care. Just being clear.
- PROTO: The protocol used by the incoming packet: TCP/UDP/ICMP
- DPT: The DestinationPort on your VM where the packet wants to arrive. For instance, if "DPT=22" then it's probably a packet for the SSH service.
What to do with these packet logs?
Don't ask me?? It's not like I can explicitly tell you if the TCP packet on port 1234 _from _82.242.156.152 is legit!
BUT
I can tell you how you should proceed next…
For each log line check if a program is listening on your VM on this specific port. Looking for _1234 _port :
netstat -tupln | grep 1234
tcp 0 0 0.0.0.0:1234 0.0.0.0:* LISTEN 14182/random-software
There you go. A program called "random-software" (pid 14182) is listening on this port. Is this legit? You should check with your boss but if something is already listening on the VM you should probably ACCEPT the packet.
If it's legit then add a new ACCEPT rule like this:
iptables -A KUBE-PROTECTION -p tcp -m tcp --dport 1234 -j ACCEPT -m comment --comment "random-software"
On the contrary, if no processes are listening on this port then you really should investigate the IP address from which it's coming. It will help you determine why it's contacting you again and again. If it doesn't seem legit then do nothing! When we go live and activate the "default deny" it will be blocked!
Allowing only certain sources
This part is for more advanced users, consider skipping ahead for now! The "-s" option of iptables allows packets to match rules only if they come from specific IP sources:
iptables ... -s 10.1.2.3,10.4.5.6,...
This is useful to restrict access to the Kubernetes-related ports to only the Kubernetes VMs themselves.
It's a bit tricky if you want to do some fine-grained work. But you can at least rewrite the policies using the IPs of all your masters, worker nodes, and ETCDs specifying all their IP addresses. This way only the cluster components can talk to each other and random external IPs won't.
However, this will need some level of automation on your side. Because when the cluster expands or shrinks (node-wise) then you must update all the iptables "-s" list with the new set of authorized IPs.
Ansible could help you resync rules for instance. 😉
A little upstart
To get started, here is the minimum list of rules based on the Kubernetes documentation and experimentations:
- Master nodes:
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 6443 -j ACCEPT -m comment --comment "API-server"
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 10250 -j ACCEPT -m comment --comment "Kubelet"
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 30000:32767 -j ACCEPT -m comment --comment "NodePort"
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 9200 -j ACCEPT -m comment --comment "Node Exporter"
# Common network virt protocol for pod to pod communication
iptables -A KUBE-PROTECTION -i eth0 -p udp --dport 8472 -j ACCEPT -m comment --comment "VXLAN overlay"
# If using cilium
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 4240 -j ACCEPT -m comment --comment "Cilium"
# If using Flannel
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 8285 -j ACCEPT -m comment --comment "Flannel"
# Logging packets that did not match any previous rules
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6
Worker nodes:
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 10250 -j ACCEPT -m comment --comment "Kubelet"
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 30000:32767 -j ACCEPT -m comment --comment "NodePort"
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 53 -j ACCEPT -m comment --comment "CoreDNS"
iptables -A KUBE-PROTECTION -i eth0 -p udp --dport 53 -j ACCEPT -m comment --comment "CoreDNS"
# Logging packets that did not match any previous rules
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6
ETCDs:
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 2379:2380 -j ACCEPT -m comment --comment "ETCD"
# Logging packets that did not match any previous rules
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6
⚠️Remember to update "eth0" with your public interface.
Also, you can have fun with the "-s <IPs>" if you wish.
Finally, you must monitor the remaining packets and decide what to do with them. Accept? Reject? Do nothing (same as reject when we activate default deny)?
Changing the default ACCEPT to default DENY
It's time!
You have allowed all Kubernetes ports and other proprietary stuff that runs on the VMs. The rest must die!
Time to reject all that doesn't match our rules by activating the great denier !! (kinda scary)
iptables -P INPUT DROP
This means "When a packet reaches the end of the INPUT chain without having matched any rules then DROP it. It dies… Astalavista el packet ! Adios!
The default should shift from:
$ iptables -xvnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
(See "policy ACCEPT" ?)
To:
$ iptables -xvnL
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
(See "policy DROP" ?)
ℹ️ Remember that if all goes to sh*t you can just reboot the VM. You could also revert the default policy with iptables -P INPUT ACCEPT if needed.
I also recommend that you do this on one VM at a time. Start with a worker node. If all goes well then do all the workers. Then one ETCD, all ETCD, and finally one master, all masters. If the cluster is still alive then GG!
If not, then go back to your VMs and log the dropped packets with the logging command (cf above) to find what you should have left open!
IV. Keeping our iptables configuration after reboots
It's time to persist our configuration. Now this will mostly depend on your OS but I have a few suggestions.
Using a Linux package
sudo apt-get update
sudo apt-get install iptables-persistent
During the installation, you will be prompted to save the current IPv4 and IPv6 rules. Choose "Yes" to save the rules.
Usage:
The iptables-persistent package provides two commands: iptables-save and iptables-restore.
- To save the current iptables rules:
sudo iptables-save > /etc/iptables/rules.v4
sudo ip6tables-save > /etc/iptables/rules.v6
- To restore iptables rules at startup:
sudo systemctl enable netfilter-persistent
Now, on each system startup, the netfilter-persistent service will read the rules from /etc/iptables/rules.v4 for IPv4 and /etc/iptables/rules.v6 for IPv6 and apply them.
Alternatively, you can manually restore the rules:
sudo iptables-restore < /etc/iptables/rules.v4
sudo ip6tables-restore < /etc/iptables/rules.v6
Keep in mind that the exact procedure might vary slightly based on your Linux distribution, especially for non-Debian-based systems.
When the main interface goes UP
We have bound our rules to the public interface of the VM using "-i eth0" (or "ens5", etc). We can tell Linux to bring up our rules before the relevant interface is activated. This is, in my opinion, the safest way to go as there won't be any milliseconds where the VM is not firewalled during a reboot. Sometimes a millisecond is all it takes to get hacked.
- Create a shell script
/etc/firewall-start
#!/bin/bash
<Add your iptables rules here>
If you added all your custom rules in a custom chain then your script should perform the following iptables tasks:
A) Flush the chain (see iptables --flush)
B) Repopulate the content of your custom chain using the iptables rules mentioned earlier.
C) Check the existence of the rule that calls your custom chain from INPUT using the iptables -c option. If this returns true then your script doesn't need to add the jump rule as it already exists. If it returns false then add the jump instruction to the INPUT chain.
- Edit
/etc/network/interfaces
auto eth0
iface eth0 inet static
pre-up /etc/firewall-start # <= Add this line
This file will be read during startup and execute the script /etc/firewall-start which will populate your rules. Then the associated interface (_eth0 _here) will be brought up.
V. Conclusion
You made it alive. GG! 🎉
This may not be the most robust config that exists in the industry but you now have a working firewall on your cluster and that's a start! In the future, you may look at dealing with firewalls using Cilium and all the EBPF stuff that makes no sense. It will be great one day. Just not today. Today was iptables day! ^^
HF
Top comments (0)