DEV Community: Frederick Fernando

Enforcing Image Trust on Docker Containers using Notary

Frederick Fernando — Tue, 05 Oct 2021 06:42:51 +0000

Why worry about software supply chain security while deploying containers?

In the past few years, we have seen attacks such as NotPetya and Sunburst, which has shifted the industry’s focus to secure their supply chain. With the growing usage of Open source software, we inherit many third-party dependencies into our application. An upstream vulnerability in one of your dependencies can affect your application, making it susceptible to a potential compromise. A software supply chain is anything that goes into or affects your code from development, through your CI/CD pipeline, until it gets deployed into production.

The first step in getting supply chain security right is to start with visualizing the supply chain components which you use from scratch. You can use something like Dependency-Track to analyze your supply chain components. Once we have the end-to-end visibility, the next step is to build upward with carefully governed and secured access to analytics and visibility capabilities. From there, continuously monitor every layer for anomalous behavior. Every step in a supply chain should be “trustworthy” as a result of a combination of cryptographic attestation and verification. No step in the supply chain should rely on assumptions about the trustworthiness of any previous steps or outputs, trust relationships must be explicitly defined. Let’s look at some of the tools which help us with supply chain security in a cloud native environment.

We will be verifying container images using Notary. Notary uses the The Update Framework (TUF) specification for publishing and verifying content. Before we deep dive into enforcing image trust on Docker containers, let's take a quick look at both of these projects.
a quick overview before deep dive.

What is The Update Framework (TUF)

The Update Framework (TUF) aims to provide a framework (a set of libraries, file formats, and utilities) that can be used to secure new and existing software update systems. These systems can be package managers that are responsible for all of the software that is installed on a system, updaters that are only responsible for individual installed applications, or software library managers that install software that adds functionality such as plugins or programming language libraries. You can find the full specifications of TUF here.

What is Notary?

Notary is an implementation of the TUF specification. It is a tool for publishing and managing trusted collections of content. Publishers can digitally sign collections and consumers can verify the integrity and origin of content. This capability is built on a straightforward key management and signing interface to create signed collections and configure trusted publishers.

With Notary, anyone can provide trust over arbitrary collections of data. Using The Update Framework (TUF) as the underlying security framework, Notary takes care of the operations necessary to create, manage, and distribute the metadata necessary to ensure the integrity and freshness of your content. It performs signing of an image using TUF’s roles and keys hierarchy.

How to implement image trust in Docker?

We can reduce the attack surface of malicious containers running in your environment by implementing container image trust. With this, we can be sure that only the images you have signed are allowed to run in your environment, thus improving the supply chain security. Docker uses Notary for signing and verifying container images. Let us look at how to enforce container image trust using Docker.

We will be running the Notary server and Docker registry locally. We will then enable Docker content trust so that we can only pull images from the local Docker registry which are signed by the Notary server.

Steps to encforce container image trust using Docker:

Make sure you have docker and docker-compose installed on your system

Clone the Git repository

$ git clone https://github.com/theupdateframework/notary

The following command will build the Notary images

   $ cd notary
   $ docker-compose build

Run docker-compose, Notary server will be running on localhost:4443

   $ docker-compose up -d

Copy the config file and testing certs to your local Notary config directory. The config file has information about the notary server URL and the CA certificate.

   $ mkdir -p ~/.notary && cp cmd/notary/config.json cmd/notary/root-ca.crt ~/.notary

In development setup, Notary server uses self-signed certificates, this root-ca.crt is required to successfully connect to it from the client i.e. docker and notary CLI.

Run a docker-registry locally, the registry server will be running on localhost:5000

   $ docker run -d -p 5000:5000 --restart=always --name registry registry:2

Pull an image from docker.io

   $ docker pull nginx:latest

Tag the image so that we can push it to the local docker registry

   $ docker tag nginx:latest localhost:5000/nginx:latest

Add these variables to enable Docker content trust, these are read by docker CLI.

   $ export DOCKER_CONTENT_TRUST_SERVER=https://localhost:4443
   $ export DOCKER_CONTENT_TRUST=1

Login to local Docker registry with username and password as admin:admin
```
$ docker login localhost:5000
```
When you push the image to the local Docker registry, it will ask you for a passphrase for root key and repository key. You will be prompted to enter these passwords automatically. When we push the image to the private registry it is signed by the Notary server
```
$ docker push localhost:5000/nginx:latest
The push refers to repository [localhost:5000/nginx]
5b8c72934dfc: Pushed 
latest: digest: sha256:dca71257cd2e72840a21f0323234bb2e33fea6d949fa0f21c5102146f583486b size: 527
Signing and pushing trust metadata
Enter passphrase for root key with ID 9a1dd40: 
Enter passphrase for new repository key with ID 4d1832f: 
Repeat passphrase for new repository key with ID 4d1832f: 
Finished initializing "localhost:5000/nginx"
Successfully signed localhost:5000/nginx:latest
```
The root and the repository (targets) keys are created once, and stored locally on the client machine which pushes the first image to the repository. The passphrases you entered above will be required when you want to push a new tag to the localhost:5000/nginx repository. You can read more about different types of the keys involved in content trust and their management here.
Install notary cli
```
$ sudo apt install notary
```

You can verify whether the image pushed to the local registry is signed by the Notary server with this command

$ notary -s https://localhost:4443 -d ~/.docker/trust list localhost:5000/nginx

NAME    DIGEST                                                              SIZE (BYTES)    ROLE
----    ------                                                              ------------    ----
latest  2f1cd90e00fe2c991e18272bb35d6a8258eeb27785d121aa4cc1ae4235167cfd    1570            targets

The -s flag indicates the location of the Notary server. The directory specified by -d flag has all the keys which were generated in previous steps along with the cache of already downloaded trust metadata.

Let’s try to download any other image which has not been signed by notary

$ docker pull alpine:latest
Error: error contacting notary server: x509: certificate is valid for notary-server, notaryserver, localhost, not localhost

We won’t be able to pull images which aren’t signed by the Notary server as we have content trust enabled, and have successfully implemented container image trust in our environment.

State of content trust in Kubernetes

Kubernetes does not support content trust natively as of now. But there are some ways you can achieve a similar result. To implement content trust in Kubernetes, you can use a container runtime which supports content trust. Docker is the only container runtime which supports this as of now. This feature is not supported by other runtimes such as CRI-O. You can read more on this in this GitHub issue here.

Similar to the approach we used in the previous section, you can make sure the environment variables DOCKER_CONTENT_TRUST and DOCKER_CONTENT_TRUST_SERVER are set correctly on each of the worker nodes of the cluster. Each image which gets pulled on to the nodes will get verified with the Notary server before running. This approach enables content trust globally in your Kubernetes environment. It also assumes that you will be using a private container repository and will be pulling the images exclusively from this private repository.

An alternative approach is to use an admission controller in the Kubernetes cluster. This controller will intercept each workload creation request, verify if the image being used in the workload spec is signed. If it is not signed, then the request to create or update the workload will be rejected by the controller. OPA + Rego can be used to build such admission controller. These are some ways we can achieve content trust in Kubernetes environments.

We've explored the importance of supply chain security, enabling content trust in Docker using Notary and ways to implement content trust in Kubernetes. That’s all folks, we will be looking more on supply chain and cloud-native security in the upcoming posts.

References

This blog post was originally posted on InfraCloud's blog

Starting your Cloud Security Journey

Frederick Fernando — Tue, 05 Oct 2021 06:33:55 +0000

When you start building your cloud infrastructure, security might not be a top priority as much as getting your project up and running. This might lead you to delay security considerations for later and end up having an insecure cloud environment for a long period of time. The more this is put off, the more difficult and expensive it is going to be to fix this. In worst cases, it might be necessary to re-architect your environment. Worse yet if you do have a security attack, it will cost you considerably more money and effort to respond to these threats. In some cases, external incident response teams have to be brought in and making them do this work is exponentially more expensive. This blog post is to guide you through the basic steps you can take to improve your cloud security posture in the early stages of building your cloud infrastructure.

Some of the important focus areas of starting your cloud security journey are:

Threat modeling
Fixing security misconfigurations
Containing network access
Improving access to cloud resources
Securing IAM policies
Implementing logging and monitoring

This guide is focused towards AWS environments, but almost all of these implementations can be carried forward to other cloud providers. You can also find similar tools for the cloud provider of your choice.

Threat modeling

Threat modeling is the process of identifying threats and then defining countermeasures to prevent or mitigate the threats to the system. This works for a wide range of systems including cloud architectures, applications, IoT devices and business processes.

Steps involved in threat modeling:

Build an architecture diagram
Identify threats
Mitigate
Validate

The first step involved in threat modelling is to build an architecture diagram of the system you are protecting. This also involves building a list of assets in the system. You cannot protect something which you don’t have visibility over. We can use tools such as CloudMapper from the previous step for building an architecture diagram of an AWS cloud environment. There are similar tools for threat modeling for other cloud service providers like Cartography.

Once you have an architecture diagram and inventory list, you can start identifying threats which can occur to the system. You can use threat modeling methods like STRIDE or PASTA to achieve this.

An example of this would be: What is the likelihood of an attack if a developer's cloud credentials were to be leaked? What are the mitigating controls to minimize damage if the credentials were to be compromised?

Take action by implementing the mitigating controls. This would help you architect a system keeping the security of it as a crucial part of the design process.

Fixing security misconfigurations

One of the major cloud vulnerabilities are security misconfigurations. According to this report, over 95% of the cloud vulnerabilities are related to cloud misconfigurations. Similar to the previous challenge, having visibility into your current cloud environment is quite difficult. Common misconfigurations include having overly permissive firewall rules/security groups.

Prowler

Prowler is an open source security auditing tool which can help us identify cloud security misconfigurations. Prowler is AWS only. Prowler is a command line tool that helps you with AWS security assessment, auditing, hardening, and incident response. It follows CIS benchmarks for AWS and can also assist in GDPR, HIPAA, PCI-DSS, ISO 27001, SOC2, and others.

You can find the Prowler installation steps here.

Prowler will give you a list of security vulnerabilities in your cloud account. You can take an export of the findings in HTML, CSV, JSON or json-asff format.

Integration with Security Hub

Prowler integrates natively with AWS Security Hub. With Security Hub, you have a single place that aggregates, organizes, and prioritizes the security alerts or findings from multiple AWS services such as AWS GuardDuty, Inspector, Macie and others. The json-asff format comes in handy for integration with AWS Security Hub.

Containing network access

Building a network diagram

We need a network diagram to assess the architecture of the cloud environment. It might be difficult for teams to have an updated network diagram, and due to the dynamic nature of the cloud, you might not know what your cloud environment looks like at any moment. Cloud resources are more often than not spun up without visibility and it gets difficult to have a controlled environment.

For these reasons, it is good to use tools like CloudMapper to visualize the current layout of your cloud environment.

CloudMapper

CloudMapper helps you analyze your Amazon Web Services (AWS) environments. It aids in building network diagrams and displays them in your browser. You can find the CloudMapper setup steps here
and using a Docker container.
Make sure that you create required read only IAM roles with these privileges.

arn:aws:iam::aws:policy/SecurityAudit
arn:aws:iam::aws:policy/job-function/ViewOnlyAccess

Network segmentation

By implementing network segmentation we reduce the network attack surface. Consider adopting a fail safe network access approach rather than fail open. Explicit network access should be allowed for required applications. You can implement micro segmentation by using Kubernetes network policies for container workloads. Different environments like staging, development and production should reside in segmented networks or different accounts. Different cloud accounts for business units, workloads, logging, monitoring, identity etc. should be used to further reduce the attack surface.

Improving access to cloud resources

Connectivity to your cloud resources should have secure and private access. There can be multiple ways to achieve this, through a VPN setup to a bastion/jump box. Or have a central authentication system with something like Keycloak or Teleport. You can integrate these systems with federated authentication mechanisms similar to Okta or any other SSO implementations.
Another way of doing this is by using AWS SSM, this takes away the overhead of managing your own central authentication system.

Securing IAM policies

User access policies (IAM policies) is another core control which can help you secure your cloud infrastructure. By having good security hygiene around IAM policies you can reduce the potential of an attacker doing extensive damage in case of a breach. Here are some of the minimal security controls for a new project.

Some IAM controls you should consider implementing:

Lock away your cloud account root user access keys
Create individual IAM users
Use user groups to assign permissions to IAM users
Grant least privilege
Configure a strong password policy for your users
Enable MFA
Use roles to delegate permissions
Do not share access keys
Rotate credentials regularly
Remove unnecessary credentials
Monitor activity in your cloud account

You can use tools like policy_sentry to create least privilege IAM policies.

Security logging and monitoring

Security logging and monitoring is an essential part of your cloud security program. This will help you identify what went wrong in your cloud environment and is the only way you can uncover malicious events.

For starters, you can at least have Cloudtrail enabled in all regions and put it in cloud storage. Next, you should set alerts for common security use cases, like multiple authentication denies or privilege escalation failures.

What else to log?

API Call Logs (Cloudtrail), DNS logs (Route53), Network Access logs (VPC Flow logs), Cloud storage logs (S3), Security services logs like Security Hub, GuardDuty, WAF logs and application logs need to be logged to build detection rules.
It is recommended to have a separate logging account for long term storage and have a centralized monitoring account and shipping the data into an SIEM like Elasticsearch. Build detection rules for cloud and application security use cases here.

Conclusion

By keeping security as one of the core principles while building out your cloud infrastructure, you save time on security efforts when you have to scale your operations. By doing these steps you can start building your cloud security. Security is a hard problem for businesses to solve and it's best you get working on these challenges from the get go.

That’s all folks, lookout for more posts on security for the next steps in your cloud security journey. I’d love to hear your thoughts on this post, do start a conversation with me on Twitter or LinkedIn :).

This post was originally published on InfraCloud's blog.