DEV Community: Hugo Prudente

Managing Secrets During Docker Build

Hugo Prudente — Sat, 06 Mar 2021 01:01:52 +0000

Original post from: https://hugoprudente.github.io

What would be the best way to manage my secrets during a docker build?

Checking official and unofficial projects available in hub.docker.com,
I have collected the 4 (four) most common cases on how users are storing and managing their secrets.

There are cases that during the build you would use a token or secret file for fetch information from a
repo or other application to setup a configuration that will not be possible during runtime.

Some of those cases also doesn't fit the multistage building as fetching a package from pip.

Scenarios

I need to install a python using a private pip that I created for this lab.

To achieve that you only need to add pip.conf file as below to /root/.pip/pip.conf.

[global]
index-url = https://hugo.prudente:My$3cr3tP4$$@private.pip/playlist
timeout=60
extra-index-url = https://pypi.python.org/simple

Looks simple, let's see how we manage it.

Method 1

Here we copy the pip.conf to the container and don't remove it on the end.

FROM python:latest

COPY pip.conf /root/.pip/pip.conf

RUN pip install playlist

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker build -t secret:v1 .

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker history secret:v1
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
0d6589d4b95f   About a minute ago   RUN /bin/sh -c pip install playlist # bui…   14.1MB    buildkit.dockerfile.v0
<missing>      4 minutes ago        COPY pip.conf /root/.pip/pip.conf # buildkit    200B      buildkit.dockerfile.v0
<missing>      10 days ago          /bin/sh -c #(nop)  CMD ["python3"]              0B

Let's check if the file on the end of the build is present and it was leaked.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
❯ docker run -it secret:v1 cat /root/.pip/pip.conf
[global]
index-url = https://hugo.prudente:My$3cr3tP4$$@private.pip/playlist
timeout=60
extra-index-url = https://pypi.python.org/simple

Method 2

Here we copy the pip.conf to the container and remove it with a RUN statement on the end.

FROM python:latest

COPY pip.conf /root/.pip/pip.conf

RUN pip install playlist
RUN rm /root/.pip/pip.conf

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker build -t secret:v2 .

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker history secret:v2
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
42f04cdc6577   6 seconds ago   RUN /bin/sh -c rm /root/.pip/pip.conf # buil…   0B        buildkit.dockerfile.v0
<missing>      4 minutes ago   RUN /bin/sh -c pip install playlist # bui…   14.1MB    buildkit.dockerfile.v0
<missing>      7 minutes ago   COPY pip.conf /root/.pip/pip.conf # buildkit    200B      buildkit.dockerfile.v0
<missing>      10 days ago     /bin/sh -c #(nop)  CMD ["python3"]              0B

Let's check again if the file was present.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker run -it secret:v2 cat /root/.pip/pip.conf
cat: /root/.pip/pip.conf: No such file or directory

Method 3

Here we copy the pip.conf to the container and remove it in the same RUN statement as the pip install

FROM python:latest

COPY pip.conf /root/.pip/pip.conf

RUN pip install playlist && rm /root/.pip/pip.conf

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker build -t secret:v3 .

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker history secret:v3
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
a2bf2672abaf   About a minute ago   RUN /bin/sh -c pip install playlist && rm…   14.1MB    buildkit.dockerfile.v0
<missing>      17 minutes ago       COPY pip.conf /root/.pip/pip.conf # buildkit    200B      buildkit.dockerfile.v0
<missing>      10 days ago          /bin/sh -c #(nop)  CMD ["python3"]              0B

Let's check once more if the file was present.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker run -it secret:v3 cat /root/.pip/pip.conf
cat: /root/.pip/pip.conf: No such file or directory

Method 4

Here we create the pip.conf using the generate.sh script that receive the
SECRET as ARG with the --build-arg options and we remove it on the same RUN statement.

#!/bin/sh

SECRET=$1

mkdir -p /root/.pip
cat > /root/.pip/pip.conf << EOF
[global]
index-url = https://hugo.prudente:${SECRET}@private.pip/playlist
timeout=60
extra-index-url = https://pypi.python.org/simple
EOF

FROM python:latest

ARG SECRET

COPY generate.sh /generate.sh

RUN /generate.sh ${SECRET} && pip install playlist && rm /root/.pip/pip.conf

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker build -t secret:v4 --progress plain --build-arg SECRET=My$3cr3tP4$$ .

➜ on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker history -H secret:v4
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
d2ca3623139f   2 minutes ago   RUN |1 SECRET=My$3cr3tP4$$ /b…   14.1MB    buildkit.dockerfile.v0
<missing>      2 minutes ago   COPY generate.sh /generate.sh # buildkit        261B      buildkit.dockerfile.v0
<missing>      2 minutes ago   ARG SECRET                                      0B        buildkit.dockerfile.v0
<missing>      10 days ago     /bin/sh -c #(nop)  CMD ["python3"]              0B

Last but not least, let's check the presece of the file.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
❯ docker run -it secret:v4 cat /root/.pip/pip.conf
cat: /root/.pip/pip.conf: No such file or directory

The credential is not on the pip.conf file but it's visible during the docker history.

Preliminary Results

Here is a matrix on where our secrets have been leaked.

Method	Runtime	Inspection
secret:v1	Yes	No
secret:v2	No	No
secret:v3	No	No
secret:v4	No	Yes

With the preliminary results we can already exclude methods 1 and 4 as we can
consider them insecure due to the credentials being visible at some point.

The method 4, I also used --build-args SECRET=${SECRET} and the secrets
leaks on the same way.

Dive Deep

OverlayFS

Is the kernel implementation for a union-filesystem, an overlay-filesystem tries
to present a filesystem which is the result over overlaying one filesystem on
top of the other.

In short taking the example of the image above imagine you have 2 directories
the lower and upper where the lower is a read-only directory for the
consumer, but they are still read-write from the Linux Operational system.

When a file is modified on the upper the change happens normally but if a
change is made on a file of the lower directory a copy of it is created on the
upper to become accessible, once the modification is complete another
process is responsible to fetch the modification and write on the lower
directory.

So this union of directories merged together as one unique block limited by the
cgroups is what docker and it's storage driver uses on our example.

The AUFS that's also a union-filesystem also presents the same behaviour,
although some of locations may be slightly different.

Checking the Filesystem

Now that I know how OverlayFS works let's isolate the directories (layers) used
by the python:latest so we can filter out only the ones that we are interested
on, the ones that have pip.conf file.

Inspect the containers I can find the directories used on the OverlayFS.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker inspect python:latest | grep Dir | grep -Eo "([a-z0-9]{25,64})"
9e7c768dda91c4fa7ed6a57c7cb784834033bff92bd11ff6d062d4de11c0f898
17bf53c98685ae36487eb55f0d2256d168f210a688ef51deef760de1a699cbdf
4cb1dbbf58a2b1ca8df6d9d977a66fe918aee21434fcd656f1a68f1f412d75ff
358dd0944f115e2a273c5259dd1432b44e36908cf223f8ce0d9f74550430f577
c034592b1a26552525742ed81e7fbce2139817b634d48db8349dbebf15a45914
19d471e0407c0f1ca14eb1cb8c46aaef9357037cad5dc170cb6a4af3c1feab40
e005796f193e62e9db78de1df20999daca1a96a0bebed19c1dd906b1b4da8542
badc6aa65b2d3f10b0cdff3fc04bf3a64b551af1dd9e01b6ecd38ed71abdc3da
8d6ff96b718838005288a94cdc9fd408d1f70d7e9cbab678ebeb4521d11b366d

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker inspect python:latest | grep Dir | grep -Eo "([a-z0-9]{25,64})" > layers.python

With the layers saved in the file layers.python I can use a similar command to
exclude the know python layers and get only the ones added by our build.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker inspect secret:v1 | grep Dir | grep -Eo "([a-z0-9]{25,64})" | grep -v -f layers.python  | uniq
e1kq2j71b7clcwtn0lbmqa1g9
v6zy2xgzrow2mgpyq9d0vch6l

➜ docker inspect secret:v2 | grep Dir | grep -Eo "([a-z0-9]{25,64})" | grep -v -f layers.python  | uniq
v6zy2xgzrow2mgpyq9d0vch6l
e1kq2j71b7clcwtn0lbmqa1g9
oudofb9c0iaqog9sff81f8053

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker inspect secret:v3 | grep Dir | grep -Eo "([a-z0-9]{25,64})" | grep -v -f layers.python  | uniq
e1kq2j71b7clcwtn0lbmqa1g9
hqsrze873a2uz7tjsgbqdo3sd

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker inspect secret:v4 | grep Dir | grep -Eo "([a-z0-9]{25,64})" | grep -v -f layers.python  | uniq
78gz619okusgrq4jo4ek863ug
oq22x88p26hzqmr1w1qpf8mm4

Now that we have the layers I have created a list just to make it simpler to
find it when we check the directory.

Accessing the OverlayFS directories

Using one of the commands below as root you will find the entry point where
Docker Storage driver creates the file-system hierarchy used by the whole
eco-system.

Linux

cd /var/lib/docker/

MacOS

docker run -it --rm --privileged --pid=host justincormack/nsenter1
cd /var/lib/docker/

Once in the /var/lib/docker directory using a simple ls and filtering the
layers previous stored in the temporary file and expanding it I could find the
specific layers that have the pip.conf

/var/lib/docker/overlay2 
# ls | grep -f /tmp/layers | xargs find | grep pip.conf
e1kq2j71b7clcwtn0lbmqa1g9/diff/root/.pip/pip.conf
hqsrze873a2uz7tjsgbqdo3sd/diff/root/.pip/pip.conf
oudofb9c0iaqog9sff81f8053/diff/root/.pip/pip.conf

So I accessed each of of those to confirm if the file was present or if was just
its shadow left by the directory union.

/var/lib/docker/overlay2 
# cat e1kq2j71b7clcwtn0lbmqa1g9/diff/root/.pip/pip.conf
[global]
index-url = https://hugo.prudente:My$3cr3tP4$$@private.pip/playlist
timeout=60
extra-index-url = https://pypi.python.org/simpl

/var/lib/docker/overlay2 
# cat hqsrze873a2uz7tjsgbqdo3sd/diff/root/.pip/pip.conf
cat: can\'t open 'hqsrze873a2uz7tjsgbqdo3sd/diff/root/.pip/pip.conf': No such device or address

/var/lib/docker/overlay2 
# cat oudofb9c0iaqog9sff81f8053/diff/root/.pip/pip.conf
cat: can\'t open 'oudofb9c0iaqog9sff81f8053/diff/root/.pip/pip.conf': No such device or address

So 1 of 3 layers have the file present so I have checked from which container
that layer belong to and here's the surprise.

That 1 layer is shared with 3 of 4 builds that we have created, meaning that
during a docker pull three diferente containers could leak my pip.conf
secret.

Results

The updated matrix consolidating the results on where our secrets have leaked.

Method	Runtime	Inspection	OverlayFS
secret:v1	Yes	No	Yes
secret:v2	No	No	Yes
secret:v3	No	No	Yes
secret:v4	NO	Yes	No

So even knowing that the file is not acessible from the container directly if
you have access to pull the container on a full read-write system you would be
able to retreive the secrets.

But now what's the best way to build the container and do not have such issue?

Solution

From 18.09 or newer Docker have introduced the Docker BuildKit that brings some
extra funcionality to the Docker builds.

The builds using BuildKit different from the legacy allows the usage of the
--secret that allows the capacity of binding a file during build runtime
similar to the tradicional runtime that we achieve with -v option.

It's usage is quite simple let's build a container and run our tests again.

FROM python:latest

RUN --mount=type=secret,id=pip.conf,dst=/root/.pip/pip.conf \
      pip install playlist

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker build --file Dockerfile  --secret id=pip.conf,src=pip.conf -t secret:v5 .

Now that we have the secret:v5 build lets confirm.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
➜ docker history secret:v5
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
266a21bb36ae   36 seconds ago   RUN /bin/sh -c pip install playlist # bui…   14.1MB    buildkit.dockerfile.v0
<missing>      11 days ago      /bin/sh -c #(nop)  CMD ["python3"]              0Bi

The history in this case is clean, not even mention the mount for pip.conf

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
❯ docker run -it secret:v5 cat /root/.pip/pip.conf
cat: /root/.pip/pip.conf: No such file or directory

The file is also not present on the system.

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
❯ docker inspect secret:v5 | grep Dir | grep -Eo "([a-z0-9]{25,64})" | grep -v -f layers.python  | uniq
cnpw0dw9o05lmdz3j9j62jzpt

on ⛵ k3s (nerdweek) ~/post via 🐍 v3.9.1 (osx)
/var/lib/docker/overlay2 # 
ls cnpw0dw9o05lmdz3j9j62jzpt | xargs find | grep pip.conf
find: committed: No such file or directory
find: diff: No such file or directory
find: link: No such file or directory
find: lower: No such file or directory
find: work: No such file or directory

And the most important one the file doesn't exist on the layer/directory that we just
created meaning that if we use in a base image our scretes are safe.

References

Using AWS Network ACLs With NAT Gateway

Hugo Prudente — Sun, 29 Nov 2020 23:34:33 +0000

It’s quite common the mistakes made when using the AWS Network ACLs for adding that extra layer of security in your VPC.

Given the fact that Network ACLs are stateless, meaning that the Inbound (Ingress) should have a matching rule for Outbound (Egress).

With this post, you will learn how to identify such particularities using an AWS ElasticBeanstalk environment as an example, due to its sensitivity regarding network access requirements.

Common Issues

Fail to access any HTTPS/TLS endpoint resulting in timeout
Fail to sync NTP servers

Difference between Security Group and Network ACL (NACL)

The main difference between the Security Group and the Network ACL (NACL) is the
the context where they are applied and the type of rules they provided.

A Security Group:
Stateful: Therefore you don't need a rule that allows response traffic for inbound requests.
Local: Therefore it applies only to the instance or service to which the security group is attached to.
A Network ACL:
Stateless: Therefore this rule is required to allow response traffic for inbound requests on the outbound rules.
Global: Therefore it applies to all services that are placed on the subnet that is attached to.

Scenario 1 - (Web Tier) Public Subnet with Network ACL without AWS NAT Gateway.

Using the diagram below as an example to configure the Network ACL's for an ElasticBeanstalk environment.

Diagram:

+------------------+
|                  |
|   +----------+   |
|   | INSTANCE |   |
|   +----------+   |
|   |  SG-001  |   |
|   +----------+   |
+------------------+
|    Subnet Pub    |
+------------------+
         +   +--------------+
         |---| ACL-001      |
         v   +--------------+
+------------------+
|       IGW        |
+------------------+

As presented on the common issues, the simplified flow for UDP and HTTPS TCPDump packet below displays the return of 2 (two) actions from the EC2 belonging to the ElasticBeanstalk test environment.

UDP 172.31.0.31:123 > 0.amazon.pool.ntp.org
UDP 0.amazon.pool.ntp.org > 172.31.0.31:2000

TCP 172.31.0.31:443 > aws.amazon.com
TCP aws.amazon.com > 172.31.0.31:3000

As we notice from the packet analysis the returning port is ephemeral and as we already know that Network ACL are stateless we need to account for that during the rules creation. Due to the Security Group being stateful, the above behavior will work properly.

Solution

The solution, in this case, is a mix between the Security Group (SG-001) and the Network ACLs (ACL-001) where we can see:

Security Group presents inbound traffic for TCP/HTTP, TCP/HTTPS for standard ports 80 and 443 from everywhere and outbound traffic from the instance to everywhere.

Security Group Inbound Rule (SG-001)

Type  |Protocol |Port Range |Source
--------|-----------|-----------|------
HTTP  |TCP  |80 |0.0.0.0/0
HTTPS |TCP  |443  |0.0.0.0/0

Security Group Outbound Rule (SG-001)

Type  |Protocol |Port Range |Source
--------|-----------|-----------|------
All |Traffic  |ALL  |ALL  |0.0.0.0/0

Network ACL Inbound (ACL-001)

Rule #  Type  Protocol  Port Range  Destination Allow / Deny
102 HTTP (80) TCP (6) 80  0.0.0.0/0 ALLOW
103 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW
201 Custom TCP Rule TCP (6) 1024-65535  0.0.0.0/0 ALLOW
202 Custom UDP Rule UDP (17)  123 0.0.0.0/0 ALLOW
* ALL Traffic ALL ALL 0.0.0.0/0 DENY

Network ACL Outbond (ACL-001)

Rule #  Type  Protocol  Port Range  Destination Allow / Deny
101 HTTP (80) TCP (6) 80  0.0.0.0/0 ALLOW
202 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW
102 Custom TCP Rule TCP (6) 1024-65535  0.0.0.0/0 ALLOW
103 Custom UDP Rule)  UDP (17)  1024-65535  0.0.0.0/0 ALLOW
* ALL Traffic ALL ALL 0.0.0.0/0 DENY

As we know that ElasticBeanstalk does more than serve traffic but also has agents that connect to AWS API's we notice two different sets of configurations in the Network ACL's.

Rules 102 and 103 allow inbound traffic for ports 80 and 443 following by its outbound pair, the 102 and 103 that are responsible to allow ephemeral port out to answer the requests.
Rules 202 and 203 are using in the reverse order, an agent will post from the EC2 to AWS API, we analyze it inverting the tables. In this case, outbound requires 80 and 443 for the request, and the response will return in an ephemeral port on the inbound rules.

Scenario 2 - (Worker Tier) Private Subnet with Network ACL and AWS NAT Gateway.

Again using the diagram below as an example to configure the Network ACL's for an ElasticBeanstalk environment.

Diagram:

+------------------+
|                  |
|   +----------+   |
|   | INSTANCE |   |
|   +----------+   |
|   |  SG-002  |   |
|   +----------+   |
+------------------+
|  Subnet Private  |
+------------------+
          +
          |   +--------------+
          |---| ACL-002      |
          v   +--------------+
+------------------+
|                  |
|   +----------+   |
|   |  NAT GW  |   |
|   |  SG-002  |   |
|   +----------+   |
+------------------+
|    Subnet Pub    |
+------------------+
         +   +--------------+
         |---| ACL-003      |
         v   +--------------+
+------------------+
|       IGW        |
+------------------+

Same as before as presented on the common issues, the simplified flow for UDP and HTTPS TCPDump packet below displays the return of 2 (two) actions from the EC2 belonging to the ElasticBeanstalk test environment.

UDP 172.31.0.31:123 > 172.31.0.34:2000
UDP 172.31.0.34:2000 > 0.amazon.pool.ntp.org
UDP 0.amazon.pool.ntp.org > 172.31.0.34:2000
UDP 172.31.0.34:2000 > 172.31.0.31:3000

TCP 172.31.0.31:443 > 172.31.0.33:2000
TCP 172.31.0.33:2000 > aws.amazon.com
TCP aws.amazon.com > 172.31.0.33:3000
TCP 172.31.0.33:3000 > 172.31.0.31:3000

We can see that this is a little bit different than before, now 4 (four) pairs of packets with a extra jump returning different ports for each pair.

It happens due to a particular behavior of the AWS NAT Gateway. AWS Nat Gateway encapsulates packets to achieve higher performance, causing the change on the packet header during its life cycle.

Solution

The solution, in this case, is a also a mix between the Security Group (SG-002) and the Network ACLs but this case ACL-002 and ACL-003 where we can see:

Security Group Inbound Rule (SG-002)

Type  Protocol  Port Range  Source
HTTP  TCP 80  0.0.0.0/0
Custom UDP  UDP 123 0.0.0.0/0
Custom UDP  UDP 1024-65535  0.0.0.0/0
Security Group Outbond Rule (SG-002)
Type  Protocol  Port Range  Source
All Traffic ALL ALL 0.0.0.0/0

Network ACL Inbound - Private Subnet (ACL-002)

Rule #  Type  Protocol  Port Range  Destination Allow / Deny
102 HTTP (80) TCP (6) 80  0.0.0.0/0 ALLOW
103 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW
202 Custom TCP Rule TCP (6) 1024-65535  0.0.0.0/0 ALLOW
203 Custom UDP Rule UDP (17)  123 0.0.0.0/0 ALLOW
* ALL Traffic ALL ALL 0.0.0.0/0 DENY

Network ACL Outbond - Private Subnet (ACL-002)

Rule #  Type  Protocol  Port Range  Destination Allow / Deny
202 HTTP (80) TCP (6) 80  0.0.0.0/0 ALLOW
203 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW
102 Custom TCP Rule TCP (6) 1024-65535  0.0.0.0/0 ALLOW
103 Custom UDP Rule)  UDP (17)  1024-65535  0.0.0.0/0 ALLOW
* ALL Traffic ALL ALL 0.0.0.0/0 DENY

Network ACL Inbound - Public Subnet (ACL-003)

Rule #  Type  Protocol  Port Range  Destination Allow / Deny
102 HTTP (80) TCP (6) 80  0.0.0.0/0 ALLOW
103 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW
202 Custom TCP Rule TCP (6) 1024-65535  0.0.0.0/0 ALLOW
203 Custom UDP Rule UDP (17)  123 0.0.0.0/0 ALLOW
* ALL Traffic ALL ALL 0.0.0.0/0 DENY

Network ACL Outbond - Public Subnet (ACL-003)

Rule #  Type  Protocol  Port Range  Destination Allow / Deny
202 HTTP (80) TCP (6) 80  0.0.0.0/0 ALLOW
203 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW
102 Custom TCP Rule TCP (6) 1024-65535  0.0.0.0/0 ALLOW
103 Custom UDP Rule)  UDP (17)  1024-65535  0.0.0.0/0 ALLOW
* ALL Traffic ALL ALL 0.0.0.0/0 DENY

Again we have the rule set for the Network ACL where.

ACL-002 Rules 102 and 103 allow inbound traffic for ports 80 and 443 following by its outbound pair, the 102 and 103 that are responsible to allow ephemeral port out to answer the requests.
ACL-002 Rules 202 and 203 are using in the reverse order, an agent will post from the EC2 to AWS API, we analyze it inverting the tables. In this case, outbound requires 80 and 443 for the request, and the response will return in an ephemeral port on the inbound rules.
We would expect only outbound traffic for the NAT Gateway, but due to the encapsulation, NAT Gateway is opening a new TCP connection with the backend using the new port requiring the same set as before.

These problems only happen if you tight the security on the Network ACL used by the AWS NAT Gateway, as the AWS default NACL that allows all the traffic inbound and outbound.

Originally posted on: hugoprudente.github.io/en/posts/

References: