DEV Community: Tiexin Guo

Container Security Scanning: Vulnerabilities, Risks and Tooling

Tiexin Guo — Mon, 07 Oct 2024 15:21:27 +0000

Over the past decade, the rise of microservice architectures, the DevOps culture, and popular tools like Docker and Kubernetes have made deploying applications as containers standard practice in the industry, making container security a top priority.

In this article, we will take a deep look at container security: what are the common container vulnerabilities, how to mitigate risks by container security scanning, and what popular container security tools and solutions are available to integrate into the SDLC in a DevSecOps way. Without further ado, let's get started.

Container Vulnerabilities

Containers are created using container images, which act as templates. If the images have vulnerabilities, the containers will introduce those vulnerabilities into the production environment.

Container images have a collection of files in them: source code, but also configurations, binaries, and other dependencies, all of which could introduce vulnerabilities. So, what are the common container vulnerabilities? Here are a few:

Misconfigurations and hard-coded secrets.
Vulnerabilities in the application code.
Vulnerabilities brought by insecure libraries or other dependencies that are imported into the images (for example, malicious code could be introduced by a software supply chain attack).

Container Security Scanning

Fortunately, most container vulnerabilities can be mitigated relatively easily via container security scanning, a process that analyzes images and containers for security issues. Continuous container security scanning, or continuous container security, if you will, is a critical part of DevSecOps.

How does container security scanning work? Typically, there are two different methods:

by analyzing files, packages, and dependencies. For example, by looking at insecure coding practices or misconfiguration, search for patterns that look like hard-coded secrets, and scan the application's dependencies (including third-party libraries and frameworks) against databases of known vulnerabilities.
by analyzing containers' activity. Unexpected network traffic or a a shell spawned in a container with an attached terminal are both suspicious activities that could cause security concerns.

Next, let's examine these container vulnerabilities—secrets, misconfiguration, vulnerabilities from code, and dependencies—and how different approaches and tools can help mitigate these risks.

Hard-Coded Secrets and Misconfigurations in Docker Images

Misconfigurations and hard-coded secrets reduce security by making the application and potentially the entire infrastructure vulnerable to easy exploits. Attackers can gain access to critical systems and may even escape from within containers to the host machine.

So, at a bare minimum, we need to check for misconfigurations and ensure no hard-coded secrets are present in Docker images.

💡

*Hard-coded secrets are fundamentally different from other vulnerabilities*: any exposed secret can lead to a security breach, regardless of whether the container is unused or the image is old. A Docker image is constructed from stacked layers that form its current state. This layered structure is prone to leaks because a layer can hide secrets from previous layers, making them invisible in the final state but still present within the image. So using a scanner that can thoroughly examine each layer is crucial for identifying and mitigating these hidden secrets.

Now, let's have a look at a few popular tools that can detect hard-coded secrets and misconfiguration.

ggshield

ggshield (requires a GitGuardian account) is a CLI to find more than 350+ types of hardcoded secrets (specific and generic ones) and 70+ Infrastructure-as-Code security misconfigurations.

Simply install it with your package manager (for example, for macOS, you can install ggshield using Homebrew: brew install gitguardian/tap/ggshield), and then the ggshield secret scan docker command is ready to be used to scan local docker images for secrets present in the image's creation process (Dockerfile and build arguments), and in the image's layers' filesystem.

Here's a demo of how it works:

ggshield provides many more functionalities than simply scanning for secrets. For more details, refer to the cheat sheet below (click to get the PDF) or the official documentation here

SecretScanner

Another option is SecretScanner from Deepfence. It is a standalone, open-source tool that retrieves and searches container and host filesystems, matching the contents against approximately 140 secret types (specific only).

Using SecretScanner is simple: pull the docker image for it by running docker pull quay.io/deepfenceio/deepfence_secret_scanner_ce:2.2.0, and we can simply do a docker run to scan container images.

See it in action below:

trivy

Yet another option is trivy , which is a versatile security scanner that can scan for both secrets (but only specific ones) and misconfiguration in container images, file systems, remote git repositories, and more (will touch on this a bit more in the next section). For now, let's focus on its capabilities of secrets and misconfiguration detection.

It can be installed using your package manager (for example, for macOS, you can also use Homebrew to install it: brew install trivy), and you can use it to scan both Docker images and local file systems.

For example, to scan secrets in an image, run trivy image --scanners secret IMAGE_NAME. Below is an example with a clean result:

Trivy also provides built-in checks to detect misconfigurations in popular Infrastructure as Code files, such as Docker, Kubernetes, Terraform, CloudFormation, and more. You can even define your own custom checks.

Here's an example to scan a Dockerfile locally: Given the Dockerfile below (building a simple Golang app with Alpine as the base image):

FROM alpine
WORKDIR $GOPATH/src/github.com/ironcore864/go-hello-http
COPY . .
RUN apk add git
RUN go get ./... &amp;&amp; CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o hello
CMD ["./hello"]
EXPOSE 8080/tcp

Run the following command to check for misconfigurations (as we can see, 4 are found):

Docker Image Scanning for CVEs

The previous section ensures that there are no hard-coded secrets and that the configurations are spot-on. Next, let's move on to the other files in the Docker image: OS packages, language-specific packages, dependencies, etc.

The aforementioned comprehensive scanner trivy is one of the most popular open-source security scanners. It can also detect known vulnerabilities according to the versions of installed packages. This means that everything inside the image, whether called by your code or not, is scanned.

For example, let's write a simple Python Flask application and build a Docker image out of it, then scan it with trivy.

A very simple Flask hello world app is created:

Then, we add some dependencies in requirements.txt:

requests==2.19.1
cryptography==3.3.2
flask

As you noticed, some packages are added, but not even used by our Flask app code, simply as a test.

Then we create a Dockerfile to build it:

If you wish, you can clone the repository here to get the whole code above.

We build and tag it: docker build . -t ironcore864/python-insecure-app:0.0.1, then scan it with trivy:

As you can see, multiple vulnerabilities are found, some of which are from the OS packages and some from the dependencies we added to it. Even though we do not call those packages directly with our code, since image scanning works by scanning the OS packages and dependencies—what's in the image—it finds them all.

Besides trivy, there are a few other tools worth mentioning:

clair: an open-source application for parsing image contents and reporting vulnerabilities affecting the contents, done via static analysis, not at runtime.
grype: a vulnerability scanner for container images and filesystems. It works with syft, the powerful SBOM (software bill of materials) tool for container images and filesystems. For more information about SBOMs:

Why you need an SBOM (Software Bill Of Materials)

SCA: Software Composition Analysis

Next, let's move on to a slightly different approach to vulnerability scanning: Software Composition Analysis (SCA), which automatically scans and detects vulnerabilities in components and third-party libraries.

While this might sound similar to Docker image scanning, there are key differences. SCA focuses on identifying vulnerabilities in the software dependencies and libraries used within an application, ensuring that all components are secure. In contrast, as we have seen, Docker image scanning examines the entire container image, including the operating system, application code, and configurations.

Let's have a look at one of the SCA tools snyk to get a hands-on feeling about SCA. For macOS users, you can install snyk by running:

If we scan the same app from the previous section:

We should get results similar to:

As we can see, the issues detected by the Docker image scanning in the previous section were also detected by image scanning. Note, however, that image scanning found more issues.

Both Docker image scanning and SCA work by referring to known vulnerability databases, which explains the similar results.

However, whereas Docker image scanning scans everything present inside an image, including OS packages, SCA is only concerned about the dependencies declared in our project (in our case, defined in the requirements.txt), which is a more specific scope.

At the end of the day, vulnerable dependencies (in our case, requests and cryptography) are reported by both Docker image scanning and SCA, but there are key differences:

When the scan happens in the DevOps cycle: Docker image scanning scans built images. So the scan can only happen once the image is ready, in other words, towards the end of the development cycle (typically in your CI pipeline, just before deploying it in a test environment). SCA, on the other hand, scans the source code, which means SCA is implemented in the early stages of the cycle. The bottom line is that SCA allows us to discover issues earlier.
What is the real scope of the scan? As we mentioned, SCA is only concerned with how the running code could be compromised, nothing more. But this code will run in a container, which presents its own attack surface. Ultimately, the container image is the blueprint for what will run in the production environment, so it's crucial to detect both the vulnerabilities affecting the container AND the application.

Combining the above pros and cons of both Docker image scanning and SCA, we can see that they provide complementary values at different stages. They both have their place in your automated DevSecOps workflow.

Besides snyk, there is another open-source project osv-scanner that's worth a look at: It's an SCA vulnerability scanner written in Go, which uses the data provided by osv.dev

For more information on osv-scanner (and open-source security in general), read this blog:

Open-Source Software Security

Container Runtime Security

So far, we've covered both Docker image scanning and SCA, which happened at the code/image level, with which we can discover the known vulnerabilities and fix them before our apps get deployed. Next, let's cover another type of container security scanning: container runtime security.

Container runtime security, as the name suggests, happens at the runtime level: it takes proactive measures and controls to protect containerized applications during the runtime phase, detecting unexpected behaviour, configuration changes, and attacks in near real-time.

This sounds theoretical, so let's take a look at one concrete example with Falco, an open-source tool designed to detect and alert on abnormal behaviour and potential security threats in real-time.

To quickly try Falco out, prepare a K8s cluster and install it with helm:

Note: Be sure to change the last parameter from changeme to your name.

For a more detailed installation guide, see the official documentation here.

Once Falco is up and running, we can simulate suspicious actions to see if the so-called "runtime security" platform can capture them. For As a first example, let's start a container and get a shell. In the real world, an attacker with shell access means the container is already compromised, as the attacker could try all kinds of things.

Start a container by running: kubectl run alpine --image alpine -- sh -c "sleep infinity", then execute the uptimecommandinside it:

kubectl exec -it alpine – sh -c "uptime"

(well, this is a harmless command, but you get the gist).

If we check Falco's output, we should get an alert:

Falco's base rules alert you if someone runs an interactive shell into a running container. For more standard rules, see the official documentation. Also, you can write and customize Falco rules by yourself to secure your environment.

Summary

In this blog, we've seen that there are several types of container vulnerabilities: hard-coded secrets, misconfigurations, vulnerabilities in the application code, and vulnerabilities in OS packages and dependencies.

Luckily, most of them can be detected by tools fairly easily:

ggshield, SecretScanner to scan for secrets
trivy to scan for misconfigurations
trivy, clair and grype to scan Docker images for known CVEs
SCA tools like snyk and osv-scanner to analyze software composition

Besides scanning code and images, we also can enhance our production environment by using container runtime security platforms such as Falco, which detects unexpected behaviour and attacks in real-time with predefined and customizable rules.

As a closing thought, it's worth pointing out that container security scanning has limitations: It's not great for detecting unknown vulnerabilities. If some vulnerabilities are not yet publicly disclosed, or if a new type of attacking behavior just emerged and it's not in the runtime security platform's rule list, they can't be detected.

So, although container security scanning is necessary to secure production environments, it's important to remember that it is one layer of security and not an end-game solution to mitigate all types of threats.

As a follow-up reading, you can refer to the Docker security cheat sheet, to help you understand how to run containers with the right security guardrails from the onset:

Docker Security Best Practices: Cheat Sheet

I hope you enjoyed this article. See you in the next piece!

How to Handle Secrets in Jupyter Notebooks

Tiexin Guo — Wed, 10 Jul 2024 12:37:00 +0000

With the rise of big data and machine learning, project Jupyter is becoming increasingly popular among data scientists and machine learning engineers. Jupyter Notebooks, together with IPython, provide an interactive workflow for developing, visualizing data, and writing texts and documentation, all in a single place and stored as a single document.

However, data science and machine learning projects often need to access third-party APIs, read data from a data store, or interact with cloud services. This means that, just like normal code, the code in Jupyter Notebooks also needs to use secrets and credentials.

These notebooks are nothing more than code, and, as you know, storing plaintext credentials in code is not acceptable. If you don't know why, I encourage you to have a look here to understand why it's urgent to deal with your hard-coded secrets, as well as this cheat sheet for managing and storing secrets in Git repositories.

Today, we will look at different ways of securely handling secrets in Jupyter notebooks. Whether you are using Jupyter Notebook, JupyterLab, or JupyterHub, these methods work across them all. Without further ado, let's get started.

Entering Passwords on the Fly in Jupyter Notebooks: `getpass`

This is probably the most intuitive method for handling secrets in Jupyter notebooks: entering secret values interactively on the fly.

Luckily, we have a native solution for that: getpass. It's been part of the Python standard library for a long time, meaning that you don't have to install anything, and it's already there. The use case is simple: it prompts you for a password when you run the cell and doesn't log this password even when printed out, so it's safe even if someone else takes a glance at your screen:

The advantage is you don't have any secret stored inside the notebook, so it's safe to push your code to any git repository. Even if the code repo is compromised, no critical information will be leaked. There are, however, a couple of downsides:

you will always have to enter it manually every time you run your Jupyter notebook. As you probably know, it's not unusual to run your cells dozens of times when you are developing and debugging your code. So, this can be a major burden.
you still have to store the actual value of the password somewhere. In a local file? But how do you keep the file safe and secure? What if it's lost? Where to back it up securely? And how do you share it?

So, although getpass is a great choice for a quick start when you're just starting a simple project, but it's hardly the most convenient. Let's move on to the next choice.

Reading Secrets from a File Uploaded to JupyterHub

One of the downsides of the previous approach is that you'd have to enter the secret values manually every time you run your notebook. To avoid this, it's natural to think of a solution where secrets are stored in a file (typically, a .env environment file) and loaded from the code.

The Python module python-dotenv does just that: it reads key-value pairs from a .env file and sets them as environment variables. This helps in the development of applications following the 12-factor principles.

If you are using Jupyter Notebook or JupyterLab locally, you can create a .env file locally:

# .env example
DB_PASS=hello

Then it's simply a matter of calling load_dotenv() to read the .env values and making them accessible to your notebook code, just like accessing any environment variables with os.getenv(). Example:

Better still, if you are using JupyterHub (a SaaS version of JupyterLab for groups of users)—or any cloud-based Jupyter service— you can upload the .env file from your local computer and host them online, making it shared amongst your teammates.

To do so, first, navigate to the Jupyter Notebook interface home page. Then you can add files by clicking the "Upload" button to open the file chooser modal:

Compared to getpass, this approach eliminates the toil of entering a password manually.

However, it's still far from perfect. Using a file as a single source of trust comes with its own caveats: What if you need to update a secret in the file? What if a value is updated on a local machine but not on the online one? And how can you share the file securely? Manually creating new backups sounds like a lot of trouble.

Theoretically, we have known the solution to these problems for a long time: use a version control system like Git. But we can't store hard-coded plaintext secrets in a Git repo. Or can we? What if we encrypt the .env file? Let's move on to the next method.

Encrypt Jupyter Notebook Secrets Files in Git

We can't store a file full of secrets in a Git repo, except if the file is encrypted. Yet encrypting a file brings its own set of challenges:

Where to store the key to decrypt the file?
If there is a new team member, how to allow his key to decrypt an existing file?
If I need to update some content in the encrypted file, I need to decrypt, change, and then encrypt it. That sounds like a lot of trouble.

Luckily, there is a tool that handles this overhead for us: SOPS. Simply put, SOPS (short for Secrets OPerationS) is an open-source text file editor that encrypts/decrypts files automatically and supports various encryption methods, like GPG keys, AWS KMS, etc.

After setting up proper encryption methods, you can open an encrypted file, edit it, and then save it encrypted again with no overhead at all. It supplements the previous dotenv method because now you can store the .env file safely and securely in a Git repository with minimum operational overhead when managing the file itself.

For a detailed tutorial on how to use sops, read this tutorial here.

SOPS is a great tool, but it's not the ultimate solution. You still have to decrypt the file before loading it. Also, as your project grows larger, managing encryption keys for all team members can become an added overhead.

That said, let's move on to secret managers.

The Ultimate Solution: Access Secret Managers from Jupyter Notebooks

As we can see from the previous section, storing secrets in an encrypted file is not ideal. Managing secrets is not a trivial task: We need to store them securely and use them securely; there is encryption, there is encryption at rest, encryption in transit, and encryption end-to-end; we need to do backup-restore properly... I could go on, but you get the gist.

Secret managers come to the rescue: they act as a single source of truth to store secrets securely, and we can store, manage, and access secrets from our code. Secret managers also make sharing secrets among team members easy, because we can configure different permissions for different secrets and members.

The bottom line is that thanks to secret managers, we completely remove ourselves from the overhead of managing secrets themselves, we simply use them.

Using Secret Managers in Jupyter Notebooks

Most secret managers support different ways of creating secrets: you can do it in the web UI, in the CLI console, etc, and accessing the secrets takes only a couple of lines of code.

Take AWS Secrets Manager as an example, with a couple of lines of code, you can easily read from it:

In this way, your Jupyter notebooks contain no secret, no encryption is needed, and they can be easily shared without worrying about leaking sensitive information.

If you are running Jupyter Notebook or JupyterLab locally, you simply set up your access to AWS and that's it. Accessing other secret managers from your local machine is more or less the same.

Using Cloud-Based Jupyter Service to Access Secret Managers

If you are using JupyterHub or some cloud-based Jupyter service like AWS SageMaker, there might be some extra configuration, but it's a one-time-only job.

For example, with AWS SageMaker, you need to know the IAM role used by your SageMaker instance to set up the API key for it (which can be found in the overview of the SageMaker notebook instance of the AWS Management Console), then grant access to the AWS Secrets Manager to the SageMaker notebook role. Granted, AWS is not the easiest platform. So here are a few resources to help you navigate their services if you are new to them:

To know more about AWS IAM and security best practices, read this blog
To know more about how to manage IAM roles with Infrastructure as Code (IaC), read this tutorial and the second part of it.
To know more about handling secrets in AWS Secrets Manager, see this blog post here.

If you are using Secret Manager from Google Cloud (GCP), you need to install the Secret Manager client library for Python by running pip3 install google-cloud-secret-manager, then you can access the secrets with a simple piece of code. See an example here. To learn more about how to handle secrets with Google Cloud Secret Manager, read this blog here.

For Microsoft Azure, it's more or less the same: set up permissions, install a client library, and then some simple code. For a detailed tutorial on handling secrets with Azure Key Vault, see this blog here.

Handling Secrets in Kaggle Notebooks

Since many machine learners and data scientists compete on Kaggle, it's worth talking about how to handle secrets in Kaggle.

Kaggle has a feature that allows you to use secrets more securely: In the "Add-ons" menu, choose "Secrets", then you can add key-value pairs, and Kaggle will store these secrets for you, acting like a secret manager:

When you need to read it from your notebook, it's only a couple of lines:

Summary

OK, so far, we've covered a couple of different methods on how to handle secrets in Jupyter notebooks, from the simple, enter-on-the-fly method getpass, all the way to the full-blown enterprise-grade secret managers.

To sum up, there are a few best practices to keep in mind when handling secrets safely in Jupyter notebooks:

Instead of hard-coded secrets in the code, use the getpass library to securely prompt you on the fly, or use environment variables to access them.
Keep your secrets in a separate configuration file that is not visible to Git; if you have to store it in Git, store it encrypted, and use sops or equivalents to minimize the overhead.
To securely use and share secrets in a big project, especially when sharing secrets across teams is necessary, secret managers are worth considering.

Open-Source Software Security

Tiexin Guo — Wed, 22 May 2024 12:51:29 +0000

On March 29, which seemed to be another normal Friday, a Microsoft developer shocked the world by revealing an XZ Utils (data-compression utilities) backdoor. This backdoor could potentially enable unauthorized access via SSH and remote code execution (read the full story here).

But wait a minute, because how on earth does compression have anything to do with SSH access? Short answer: dependencies. Part of the XZ Utils is a compression library liblzma, which isn't used directly by OpenSSH, but Debian and several other distributions patch OpenSSH to support systemd notifications, and systemd links to the libsystemd C library, which depends on liblzma.

Got it? No? Don't worry, because dependency is complicated, and that's one of the reasons why the attack happened in the first place.

But wait a minute again, because how could the project maintainers miss the malicious commits in the first place? It appears this backdoor was well-planned three years ago when a user started working on open-source projects. Over the years, they gained control of the project by becoming increasingly involved, gaining trust, and pressuring the founder.

Luckily, the backdoor's blast radius wasn't huge because although XZ Utils is included in many Linux distributions, the malicious version hadn't been widely deployed; it was only present in development versions of major distributions. Still, the attack served as a timely wake-up call in the open-source security world.

So, today, let's examine open-source software security: what it is, why you should care, and how to improve it.

An Introduction to Open Source Software Security

What is Open Source Software Security

I could refer to Wikipedia for a formal definition of this subject matter, but I guess it won't interest you that much. Just in case you wonder what the definition is: "Open-source software security is the measure of assurance or guarantee in the freedom from danger and risk inherent to an open-source software system."

To explain it in my own words, open-source software security is two things:

the risks that come with third-party, open-source software;
the tools and measures taken to improve open-source software's security.

Why You Should Care about Open Source Software Security

Although the XZ Utils backdoor story sounds terrifying, many people are actually pretty confident about open-source software security for some reason.

You may think you don't use open-source software, so you don't care about its security. But the fact is, you do. Whether you like to admit it or not, as much as 90% of code in any modern software is open-source. This is especially true in cloud-native time, where almost all apps rely on some open-source components.

Or, you might be pretty confident that your applications don't have any libraries with known vulnerabilities.

I know I was, until a minute ago, because out of curiosity just now, when writing this very paragraph, I ran a check on a project I have been working on recently, only to find 6 known vulnerabilities. There was no automated check in the project to make sure there weren't any known vulnerabilities, either. God, I wish I was making this up just for the sake of this article, but unfortunately, it is very real.

And I'm not alone, because according to a survey done by Sonatype, as much as two-thirds of the surveyed users feel confident that their applications do not rely on known vulnerable libraries, despite 10% of respondents reporting their organizations had security breaches due to open source vulnerabilities in the past year.

Starting to feel a bit worried? Here are more statistics to up your anxiety levels:

1 in 8 open-source downloads has known risks.
245,000 malicious packages were discovered in 2023 alone. If you don't know that's "a lot" or "maybe not so much compared to previous years," let me put it this way: it was twice the number of previous years combined.
18.6% of open source projects across Java and JavaScript that were maintained in 2022 are no longer maintained today, just one year later. How can you trust something when it is not even actively maintained? What's more, how can you know if something you depend on is still maintained?
If you think these numbers are already frightening, think this: open source software is growing rapidly - with just Java (Maven), JavaScript (npm), and Python (PyPI) three languages/package managers, the total number of projects is 3.6 million, with 54 million different versions, requested 3.8 trillion times in 2023 alone.

OK, now that I think I have scared you enough and caught your attention, next let's take a look at open-source software dependencies, which only make the open-source software security matter worse.

The Complexity of Open Source Dependencies

As mentioned in the previous section, you only write 10% code of your application, and the other 90% are actually dependencies - open-source components. The situation is more or less the same if you take a closer look at one of your open-source dependencies: the author probably also didn't write 100% of their code but used many dependencies.

Let's say you are writing a simple Python web application using Flask. If we have a look at the dependencies of Flask itself, it depends on 6 other packages:

The list isn't so long, but it's 1 package depending on 6. And let's not forget that Flask is a micro web framework—emphasis on micro—meaning it doesn't require particular tools or libraries. The same can't be said for many other bigger frameworks or even platforms. For example, you are creating a state-of-the-art machine learning model using TensorFlow. It seems you only have TensorFlow as your only dependency, but under the hood of TensorFlow, it depends on:

You know TensorFlow depends on Keras since you use it very often. But are you even aware that Keras, in turn, depends on rich, which depends on markdown-it-py, which then depends on mdurl? Do you know what mdurl actually is? I don't if I'm being honest.

If the dependency tree looks neat, let's convert it to a graph:

There is a colloquial term in software called "dependency hell," where users' installed software packages don't work because they depend on specific versions of other software packages.

The main software relies on a multitude of large software libraries. It might depend on product A, but A relies on B to function, and B needs C to work properly. Then there are conflicting programs, such as app X requiring some lib v1, but app Y requires the same lib in v2. There could even be circular dependencies.

Now, we don't have to work these out since in most languages, such as Python, Go, JavaScript, etc., there are package managers that do this for us. But this doesn't change the fact that our dependency depends on other dependencies which are indirect dependencies to our project, and this is known as "transitive dependencies."

This could be a special concern because they are less visible to us, security tools, and audits. The nature of the complicated dependencies means that there is a chance that your project's direct dependencies are fine, but their dependencies contain CVEs or malicious code.

Although many reasons could lead to vulnerabilities in open-source software, such as inconsistent quality, unsupported code/unmaintained packages, etc., transitive dependencies by far caused the most trouble. According to the State of Dependency Management report done by Station 9, the Endor Labs research team, 95% of all vulnerabilities are found in transitive dependencies instead of direct dependencies, making it extremely difficult for developers to assess the real impact of these issues, or whether they're even reachable.

I hope the open source situation doesn't scare you away from open source and lead you to move to proprietary software. Some argue that open source software is less secure because of the inconsistent quality of different contributors, human errors, etc., but the very same reasons that cause vulnerabilities in open source can exist in proprietary software as well, and proprietary software can include vulnerabilities.

It's not all bad news about open source software, though, because according to the same State of the Software Supply Chain report done by Sonatype:

96% of downloaded vulnerable releases have a fixed version available.
For every nonoptimal component upgrade that could potentially cause security risks, there are 10 superior versions of components available.

We just need to figure out how to improve security with the right tools. So, next, let's look at some security tools that can help improve open-source software security.

Vulnerability Scanning

To improve the security of our projects, let's start with the vulnerabilities themselves. We need to know if there are any known CVEs in them before we can fix them. So, the first step is to get some visibility on the Known CVEs, and luckily, there are some very good tools for this.

One of the most popular open-source security scanners is trivy by aquasec. It's an easy-to-use and fast CLI tool written in Golang that can scan container images, file systems, remote git repositories, and more. Besides known CVEs, it can also scan for secrets and misconfigurations. For Mac users, you can simply install it by brew install trivy, then trivy fs --scanners vuln myproject/ to scan for vulnerabilities in your project. And trivy can be integrated with many popular platforms and applications, such as your CI pipelines or even with your IDE.

SBOM, and Software Composition Analysis

Since we heavily rely on dependencies, most of which are open-source components, it's important to know what dependencies are there. And, open-source packages, especially small ones, are usually maintained by a small team with only a few developers (or even a single developer), if they are maintained at all. Developers of open-source packages do not commit to maintaining the software and can decide to stop maintaining them at any time, for any reason. If it happens, there will be no one updating the package to eliminate known CVEs. Therefore, organizations must be able to inventory open-source components.

SBOM - Software Bill of Materials, comes into play: It has emerged as a critical component in software security, providing critical visibility into software components and supply chains, and helping identify and avoid vulnerabilities.

Why you need an SBOM (Software Bill Of Materials)

Trivy can also generate SBOM in different formats, and to do so, it's one simple command: trivy fs --format cyclonedx --output result.json /app/myproject. There is another CLI tool and library for generating a Software Bill of Materials from container images and filesystems, which is syft. It's easy to install (for Mac users, brew install syft), supports many ecosystems such as JavaScript, Python, Go, Ruby, etc., and can be integrated with CI. For more on SBOM, see this blog post here.

Generating SBOM alone isn't enough: we need to make sure that what's in the SBOM list (our dependencies) is OK. We need to scan the dependencies to detect vulnerabilities, and this is known as SCA—Software Composition Analysis—the process of analyzing dependencies to determine if they are affected by known security vulnerabilities.

One tool for this on the rise is OSV, a database of open-source vulnerabilities built by Google in 2021. It has a CLI tool osv-scanner that serves as the frontend to the OSV database, and you can install it with a simple command:

go install github.com/google/osv-scanner/cmd/osv-scanner@v1

Then, your project's list of dependencies and the vulnerabilities that affect them is only one command away:

osv-scanner—r path/to/your/project

OSV also provides API access where you can query vulnerabilities for a particular project at a given commit hash or version, and currently, it's not rate-limited yet.

Finally, GitGuardian recently integrated a SCA module into their code security platform. It scans your apps' dependencies to detect vulnerabilities, and allows you to prioritize incidents by criticality:

It also allows you to generate per project SBOM really easily:

Check out the complete features here!

Licensing

Despite being open-source, most open-source applications and packages come with their own usage licenses, describing how you can use the code. Risks could occur if how you use it doesn't match its intended purposes and usages, and a single dependency component could violate laws or requirements the company needs to follow.

So, understanding different types of licenses so that the code is used compliantly, and this is no trivial effort since it requires knowledge of different types of licenses and requirements of the project throughout the software development life cycle. Given the complicated nature of dependencies, licenses of some dependency components could even be incompatible with each other, making the matter even more grim.

For container license security, trivy scans any container image for license files and offers an opinionated view on the risk, and by default, trivy scans licenses for packages installed by apk, apt-get, dnf, npm, pip, gem, etc.

Another tool that can help scan for license issues is fossa. It has a free plan which is limited to 5 projects. You can install the CLI by brew update && brew install --cask fossa, getting an API key from the platform and setting it in an environment variable, then simply run fossa analyze in the root directory of your project. It shows direct and indirect licenses used in the project and flags potential issues. For example, one dependency might declare itself with one license, but it contains copied open-source code declared with another license.

PS: GitGuardian SCA also allows to easily keep an eye on the open-source licences in use in your codebases.

Conclusion

As software development increasingly relies on open-source components, understanding the complexity and potential risks becomes crucial. The transitive dependencies nature of open source presents a significant challenge because it amplifies the potential for vulnerabilities within the software supply chain.

Fortunately, various tools and techniques, such as vulnerability scanning, license analysis, and software composition analysis, are available to mitigate these risks. By proactively addressing security concerns and adopting best practices, developers and organizations can safeguard their projects and maintain the integrity of their software ecosystem.

Embracing open-source software security is not only a responsibility but also an opportunity to build trust and collaboration in the software industry.

FAQ

Which is better for security, open source or proprietary software?

Open-source software is not less secure than proprietary code, and a commercial license doesn't guarantee security: Both can include vulnerabilities leading to security issues.

With proprietary software, the only choice is to trust the vendor; with open-source projects, you should not unquestioningly trust community-validated code. And since open source is transparent about potential vulnerabilities, you can make an effort on your part and improve its security.

How do you make open-source software secure?

Scan for vulnerabilities, inventory your project, map your open source dependencies to known security vulnerabilities, and use automation to continuously monitor for new risks.

What is the difference between Software Composition Analysis (SCA) and Open Source Software (OSS)?

Open-source software (OSS) is computer software that is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software and its source code to anyone and for any purpose. Software composition analysis (SCA) is the practice of analyzing custom-built software applications to detect embedded open-source software and detect if they are up-to-date, contain vulnerabilities, or have licensing requirements.

The New Frontier in Cybersecurity: Embracing Security as Code

Tiexin Guo — Tue, 26 Dec 2023 12:37:44 +0000

Security as Code (SaC) is a term often used with DevSecOps, but what does it mean exactly? Learn best practices and key components for a more secure and efficient development process.

How We Used to Handle Security

A few years ago, I was working on a completely new project for a Fortune 500 corporation, trying to bring a brand new cloud-based web service to life simultaneously in 4 different countries in the EMEA region, which would later serve millions of users.

It took me and my team two months to handle everything: cloud infrastructure as code, state-of-the-art CI/CD workflows, containerized microservices in multiple environments, frontend distributed to CDN, and tests passing in the staging environment. We were so prepared that we could go live immediately with just one extra click of a button. And we still had a whole month before the planned release date.

I know, things looked pretty good for us; until they didn't: because it was precisely at that moment a "security guy" stepped in out of nowhere and caused us two whole weeks.

Of course, the security guy. I knew vaguely that they were from the same organization but maybe a different operational unit. I also had no idea that they were involved in this project before they showed up. But I could make a good guess: writing security reports and conducting security reviews, of course. What else could it be?

After finishing those reports and reviews, I was optimistic: "We still have plenty of time," I told myself. But it wasn't long before my thought was rendered untrue by another unexpected development: an external QA team jumped in and started security tests. Oh, by the way, to make matters worse, the security tests were manual.

It was two crazy weeks of fixing and testing and rinsing and repeating. The launch was delayed, and even months after the big-bang release, the whole team was still miserable: busy on-call, fixing issues, etc.

Later, I would continue to see many other projects like this one, and I am sure you also have similar experiences. This is actually how we used to do security. Everything is smooth until it isn't because we traditionally tend to handle the security stuff at the end of the development lifecycle, which adds cost and time to fix those discovered security issues and causes delays.

Over the years, software development has evolved to agile and automatic, but how we handle security hasn't changed much: security isn't tackled until the last minute. I keep asking myself: what could've been done differently?

Understanding DevSecOps and Security as Code

DevSecOps: Shift Security to the Left of the SDLC

Based on the experience of the project above (and many other projects), we can easily conclude why the traditional way of handling security doesn't always work:

Although security is an essential aspect of software, we put that aspect at the very end of the software development lifecycle (SDLC). When we only start handling the critical aspect at the very end, it's likely to cause delays because it might cause extra unexpected changes and even rework.
Since we tend to do security only once (at least we hope so), in the end, we usually wouldn't bother automating our security tests.

To make security great again, we make two intuitive proposals for the above problems:

Shift left: why does security work at the end of the project, risking delays and rework? To change this, we want to integrate security work into every stage of the SDLC: shifting from the end (right side) to the left (beginning of the SDLC, i.e., planning, developing, etc.), so that we can discover potential issues earlier when it's a lot easier and takes much less effort to fix or even rework.
Automation: why do we do security work manually, which is time-consuming, error-prone, and hard to repeat? Automation comes to the rescue. Instead of manually defining policies and security test cases, we take a code-based approach that can be automated and repeated easily.

If we combine the "shift left" part and the "automation" part, bam, we get DevSecOps: a practice of integrating security at every stage, throughout the SDLC to accelerate delivery via automation, collaboration, fast feedback, and incremental, iterative improvements.

What is Security as Code (SaC)?

Shifting security to the left is more of a change in the mindset; what's more important is the automation, because it's the driving force and the key to achieving a better security model: without proper automation, it's difficult, if not impossible at all, to add security checks and tests at every stage of the SDLC without introducing unnecessary costs or delays. And this is the idea of Security as Code:

Security as Code (SaC) is the practice of building and integrating security into tools and workflows by identifying places where security checks, tests, and gates may be included.

For those tests and checks to run automatically on every code commit, we should define security policies, tests, and scans as code in the pipelines. Hence, security "as code".

From the definition, we can see that Security as Code is part of DevSecOps, or rather, it's how we can achieve DevSecOps. The key differences between Security as Code/DevSecOps and the traditional way of handling security are shifting left and automation: we try to define security in the early stages of SDLC and tackle it in every stage automatically.

The Importance and Benefits of Security as Code (SaC)

The biggest benefit of Security as Code, of course, is that it's accelerating the SDLC. How so? I'll talk about it from three different standpoints:

First of all, efficiency-boosting: security requirements are defined early at the beginning of a project when shifting left, which means there won't be major rework in the late stage of the project with clearly defined requirements in the first place, and there won't be a dedicated security stage before the release. With automated tests, developers can make sure every single incremental code commit is secure.

Secondly, codified security allows repeatability, reusability, and consistency. Development velocity is increased by shorter release cycles without manual security tests; security components can be reused at other places and even in other projects; changes to security requirements can be adopted comprehensively in a "change once, apply everywhere" manner without repeated and error-prone manual labor.

Last, but not least, it saves a lot of time, resources, and even money because, with automated checks, potential vulnerabilities in the development and deployment process will be caught early on in the SDLC when remediating the issues has a much smaller footprint, cost- and labor-wise.

To learn more about how adding security into DevSecOps accelerates the SDLC in every stage read this blog here and here.

Key Components of Security as Code

The components of Security as Code for application development are automated security tests, automated security scans, automated security policies, and IaC security.

Automated security tests: automate complex and time-consuming manual tests (and even penetration tests) via automation tools and custom scripts, making sure they can be reused across different environments and projects.
Automated security scans: we can integrate security scans CI/CD pipelines so that they can be triggered automatically and can be reused across different environments and projects. We can do all kinds of scans and analyses here, for example, static code scans, dynamic analyses, and vulnerability scans against known vulnerabilities.
Automated security policies: we can define different policies as code using a variety of tools, and integrate the policy checks with our pipelines. For example, we can define access control in RBAC policies for different tools; we can enforce policies in microservices, Kubernetes, and even CI/CD pipelines (for example, with the Open Policy Agent). To know more about Policy as Code and Open Policy Agent, read this blog here.
IaC security: in modern times, we often define our infrastructure (especially cloud-based) as code (IaC) and deploy it automatically. We can use IaC to ensure the same security configs and best practices are applied across all environments, and we can use Security as Code measures to make sure the infrastructure code itself is secure. To do so, we integrate security tests and checks within the IaC pipeline, as with the ggshield security scanner for your Terraform code.

Best Practices for Security as Code

With the critical components of Security as Code sorted out, let's move on to a few best practices to follow.

Security-First Mindset for Security as Code/DevSecOps

First of all, since Security as Code and DevSecOps are all about shifting left, which is not only a change of how and when we do things, but more importantly, a change of the mindset, the very first best practice for Security as Code and DewvSecOps is to build (or rather, transition into) a security-first mindset.

At Amazon, there is a famous saying, which is "Security is job zero". Why do we say that? Since it's important, if you only start dealing with it in the end, there will be consequences. Similar to writing tests, trying to fix issues or even rework components because of security issues found at the end of a project's development lifecycle can be orders of magnitude harder compared to when the code is still fresh, the risk just introduced, and no other components relying on it yet.

Because of its importance and close relationship with other moving parts, we want to shift security to the left, and the way to achieve that is by transitioning into a security-first mindset.

If you want to know more about DevSecOps and why "adding" security into your SDLC doesn't slow down, but rather speeds up things, I've got another blog here detailing exactly that.

Code Reviews + Automated Scanning

Security as Code is all about automation, so it makes sense to start writing those automated tests as early as possible, so that they can be used at the very beginning of the SDLC, acting as checks and gates, accelerating the development process. For example, we can automate SAST/DAST (static application security testing and dynamic application security testing) with well-known tools (for example, SonarQube, Synopsys, etc.) into our CI/CD pipelines so that everything runs automatically on new commits.

One thing worth pointing out is that SAST + DAST isn't enough: while static and dynamic application tests are the cornerstones of security, there are blind spots. For example, one hard-coded secret in the code is more than enough to compromise the entire system (to know more about this topic, read this blog here).

Two approaches are recommended as complements to the automated security tests. First of all, regular code reviews help. It's always nice to have another person's input on the code change because the four-eyes principle can be useful. For more tips on conducting secure code reviews, read this blog here.

However, code reviews can only be so helpful to a certain extent, because, first of all, humans still tend to miss mistakes, and second of all, during code reviews, we mainly focus on the diffs rather than what's already in the code base.

As a complement to code reviews, having security scanning and detection in place also helps. To know more about secrets in source code and how secrets sprawl works, see this blog here.

Continuous Monitoring, Feedback Loops, Knowledge Sharing

Having automated security policies as checks can only help so much if the automated security policies themselves aren't of high quality, or worse, the results can't reach the team.

Thus, creating a feedback loop to continuously deliver the results to the developers and monitor the checks is critical, too. It's even better if the monitoring can create logs automatically and display the result in a dashboard to make sure no security risk is found, no sensitive data or secret is shared, and developers can find breaches early so that they can remediate issues early.

Knowledge sharing and continuous learning can also be helpful, allowing developers to learn best practices during the coding process. Here is a cheat sheet for developers to do security in different stages of the SDLC in the cloud-native era.

Security as Code: What You Should Keep in Mind

Besides the best practices above, there are a few other considerations and challenges when putting Security as Code into action:

First of all, we need to balance speed and security when implementing Security as Code/DevSecOps. Yes, I know, earlier that we mentioned how security is job zero and how doing DevSecOps actually speeds things up, but still, implementing Security as Code in the early stages of the SDLC still costs some time upfront, and this is one of the balances we need to consider carefully, to which, unfortunately there is no one-size-fits-all answer. In general, for big and long-lasting projects, paying those upfront costs will most likely be very beneficial in the long run, but it could be worth a second thought if it's only a one-sprint or even one-day minor task with relatively clear changes and confined attack surface.

Secondly, to effectively adopt Security as Code, the skills and gaps in the team need to be identified, and continuous knowledge sharing and learning are required. This can be challenging in the DevOps/DevSecOps/Cloud world, where things evolve fast with new tools and even new methodologies emerging from time to time. Under this circumstance, it's critical to keep up with the pace, identify what could potentially push engineering productivity and security to another level, figure out what needs to be learned because we've only got limited time and can't learn everything, and learn those highly-prioritized skills quickly.

Last but not least, keep a close eye on newly discovered security threats and changes regarding security regulations, which are also evolving with time.

Conclusions and Next Steps

Security as Code isn't just a catchphrase; it requires continuous effort to make the best out of it: a change of mindset, continuous learning, new skills, new tooling, automation, and collaboration.

As a recap, here's a list of the key components:

Automated security tests
Automated security scans
Automated security policies
IaC security

And here's a list of the important aspects when adopting it proactively:

Security-first, mindset change, shift left
Automated testing/scanning with regular code reviews
Continuous feedback and learning

At last, let's end this article with an FAQ list on Security as Code and DevSecOps.

F.A.Q.

What is Security as Code (SaC)?

Security as Code (SaC) is the practice of building and integrating security into tools and workflows by defining security policies, tests, and scans as code. It identifies places where security checks, tests, and gates may be included in pipelines without adding extra overhead.

What is the main purpose of Security as Code?

The main purpose of Security as Code is to boost the SDLC by increasing efficiency and saving time and resources while minimizing vulnerabilities and risks. This approach integrates security measures into the development process from the start, rather than adding them at the end.

What is the relationship between Security as Code and DevSecOps?

DevSecOps is achieved by shifting left and automation; Security as Code handles the automation part. Security as Code is the key to DevSecOps.

What is the difference between DevSecOps and secure coding?

DevSecOps focuses on automated security tests and checks, whereas secure coding is the practice of developing computer software in such a way that guards against the accidental introduction of security vulnerabilities. Defects, bugs, and logic flaws are consistently the primary cause of commonly exploited software vulnerabilities.

What is Infrastructure as Code (IaC) security?

IaC security uses the security as code approach to enhance infrastructure code. For example, consistent cloud security policies can be embedded into the infrastructure code itself and the pipelines to reduce security risks.

How to Handle Secrets in Helm

Tiexin Guo — Thu, 16 Nov 2023 16:17:04 +0000

Kubernetes (K8s), an open-source container orchestration system, has become the de-facto standard for running containerized workloads thanks to its scalability and resilience.

Although K8s has the capabilities to streamline deployment processes, the actual deployment of applications can be cumbersome, since deploying an app to a K8s cluster typically involves managing multiple K8s manifests (like Deployment, Service, ConfigMap, Secret, Ingress, etc.) in YAML format. This isn't ideal because it introduces additional operational overhead due to the increased number of files for one app. Moreover, it often leads to duplicated, copy-pasted sections of the same app across different environments, making it more susceptible to human errors.

Helm, a popular package manager for Kubernetes, is designed to solve these deployment issues and help us manage K8s apps.

Helm Charts and Helm Secrets

Helm provides a straightforward way to define, install, and upgrade apps on K8s clusters. It is based on reusable templates called Helm charts, which encapsulate all the necessary K8s manifests, configurations, and dependencies into one single package, making it a whole lot easier to consistently version-control, publish/share, and deploy apps.

Since most apps rely on configurations, Helm charts often rely on ConfigMaps and Secrets (for sensitive information) to pass values to the apps as environment variables, following the Twelve-Factor App methodology. However, handling secrets in Helm can be challenging due to security concerns and collaboration/access control reasons:

Security concerns: Secrets contain sensitive information such as passwords, API keys, and database credentials. Managing secrets securely is crucial to protect sensitive data from unauthorized access. Helm must ensure that secrets are properly encrypted, stored, and transmitted.
Collaboration/access control: Helm promotes collaboration among teams by sharing charts and configurations. However, if secrets are included in these shared charts, controlling access becomes challenging. Ensuring that only authorized individuals can access and modify secrets is crucial for maintaining security and compliance.

In this blog, we aim to provide comprehensive solutions to solve these challenges once and for all. Here's what you can expect:

Hands-on Tutorials: We will guide you through practical tutorials that demonstrate the usage of specialized tools such as the helm-secrets plugin and the External Secrets Operator. These tutorials will equip you with the knowledge and skills to effectively manage secrets in Helm deployments.
Integration with CI/CD: Helm secrets are also a critical aspect of CI/CD pipelines. We will explore the possibilities, pros, and cons of CI/CD integrations, making sure Helm secrets and security are handled not only in manual deployment but also in automated workflows.
Secrets Rotation: Secrets rotation is a necessary security practice. We will introduce a tool that simplifies the redeployment of apps when secrets rotation occurs.
Tool Comparison and FAQs: To assist you in selecting the right tool for your specific needs, we will list the pros and cons and a flowchart to help you decide. Additionally, we will address some frequently asked questions to further clarify any doubts you may have.

Without further ado, let's dive in and explore the solutions to revolutionize your secrets management within Helm charts.

Tutorial: The helm-secrets Plugin

As mentioned in the previous section, managing secrets for Helm charts can be challenging because Helm chart Secrets contain sensitive information, and it's difficult to control access between team members.

With that in mind, there are two (and maybe only two) approaches to managing secrets in Helm charts: either we store sensitive information in Helm charts encrypted, or we don't store them in the charts at all:

Encrypt the Secrets values in the values file of the Helm charts. This way, we ensure the chart is safe to share and check into version control without worrying about leaking sensitive information. We still need to figure out a control mechanism to share access (decrypt the values) among team members.
Do not store the Secrets values in the Helm charts at all. This way, the sensitive information isn't in the Helm charts at all, so it's completely secure. Then we need to figure out a way to actually deploy Secrets into K8s clusters.

A Quick Introduction to the helm-secrets Plugin

TL;DR: the helm-secrets plugin can work in both of the two above-mentioned methods:

encrypting sensitive information in the Helm charts and decrypting the values on the fly.
storing secrets elsewhere (like in a secrets manager) and injecting them into the Helm charts when Helm deployments happen.

To learn more about how to use major cloud providers' secrets manager services, read my blogs:

How helm-secrets works

OK, now the long version.

Helm-secrets is a Helm plugin that manages secrets.

What's a Helm plugin, then? Good question.

Helm plugins are extensions that add additional functionalities and capabilities to Helm. These can be new commands, features, or integrations. Plugins can be used to perform various tasks, such as managing secrets, encrypting values, validating charts, etc. To use a Helm plugin, we typically install it using the helm plugin install command, and then we can invoke the plugin's commands just like any other native Helm command.

With helm-secrets, the values can be stored encrypted in Helm charts. However, the plugin does not handle the encryption/decryption operations itself; it offloads and delegates the cryptographic work to another tool: SOPS

SOPS, short for Secrets OPeration*S*, is an open-source text file editor by Mozilla that encrypts/decrypts files automatically. With SOPS, when we write a file, SOPS automatically encrypts the file before saving it to the disk. For that, it uses the encryption key of our choice: this can be a PGP key, an AWS KMS key, or many others. To learn more about SOPS and its possible integrations with different encryption key service providers, read my other blog here:

A Comprehensive Guide to SOPS: Managing Your Secrets Like A Visionary, Not a Functionary

helm-secrets can also work in a "cloud" mode, where secrets are not stored in the Helm chart, but in a cloud secret manager. We then simply refer to the path of the secret in the cloud in the file, and the secret is automatically injected upon invoking helm install.

Example 1: Helm-secrets with SOPS

In this example, we will store encrypted secrets in the Helm charts. Since this relies on SOPS, we first need to install it. The easiest way to install SOPS is via brew:

For other OS users, refer to the official GitHub repo of SOPS.

Then, let's configure SOPS to use PGP keys for encryption:

If you are using another OS, for example, Linux, you can use the corresponding package manager. Most likely, this would work:

sudo apt-get install gnupg

With GnuPG, Creating a key is as simple as the following (remember to put your name as the value of KEY_NAME):

Get the GPG key fingerprint:

In the "pub" part of the output, you can get the GPG key fingerprint (in my case, it's "BE574406FE117762E9F4C8B01CB98A820DCBA0FC").

Then we need to configure SOPS to use this PGP key for encryption/decryption. To do so, create a file named .sops.yaml under your $HOME directory with the following content:

Remember to replace the key fingerprint generated in the previous step.

Installing and configuring SOPS with PGP keys is not simple; refer to my blog on SOPS for more details.

Finally, we can install helm-secrets. Click here to get the latest version (at the time of writing, the latest version is v4.5.1). Then, run the following command to install:

Let's create a secret file and name it as credentials.yaml.dec with the following content:

To encrypt this file using helm-secrets, run the following command:

If you open the generated credentials.yaml file, you will see that its content is encrypted by SOPS.

Next, we can refer to the encrypted value in the Helm chart's Secrets. Suppose we have a file named your-chart/templates/secrets.yaml with the following content:

This template will use the value from the helm-secrets encrypted credentials.yaml. When installing the Helm chart, instead of using helm install, use helm secrets install:

Remember not to submit the credentials.yaml.dec file to git repositories as it contains clear text passwords. However, the SOPS encrypted result credentials.yaml can be submitted as part of the Helm chart (although it is generally recommended not to include secret values in Helm charts). If you choose to submit encrypted values, make sure to use a private chart repository for internal use only.

In this case, we can store sensitive information in an encrypted values file. To share the encrypted values file with your team members, you can add their public key fingerprints to the SOPS configuration. This way, access control is managed by SOPS rather than helm-secrets.

SOPS supports various encryption methods, such as using AWS KMS keys for encryption and sharing access through AWS IAM policies. For more encryption methods supported by SOPS, refer to my other blog.

For more details on using helm-secrets with SOPS, refer to the official doc here.

Example 2: helm-secrets with Cloud Secrets Managers

In the previous example, we discussed how to store sensitive information in an encrypted form within the values file. However, helm-secrets offers another mode that integrates with popular cloud secrets managers like Hashicorp Vault or AWS Secrets Manager.

To enable the integration with cloud secrets managers, you need to set the environment variable HELM_SECRETS_BACKEND=vals before running Helm. This will activate the vals integration in helm-secrets:

Vals is a tool specifically designed for managing secret values in Helm. It requires cloud provider credentials to fetch secrets from the secret services. Make sure you have the necessary credentials in place before attempting to use them.

Let's assume you have a file called secrets.yaml located at your-chart/templates/secrets.yaml. Here's an example of its content:

In your values.yaml file, you can include the following snippet:

Finally, you can install everything together using the following command:

This command will inject the secret value from AWS Secrets Manager, located at "path/to/my/secret/value", into the variable "password" defined in the values file.

Simplifying Continuous Deployment Integration with helm-secrets

If you're already using SOPS, then helm-secrets is a great choice for seamless integration. It also offers cloud integrations if you prefer not to store encrypted data in values files.

While helm-secrets can be integrated with major CD tools like Argo CD, there is some operational overhead involved. This is because both SOPS and the helm-secrets plugin are required by the CD tool, as shown in the previous examples.

For instance, to integrate Argo CD with helm-secrets, you need to ensure that the Argo CD server container has both SOPS and helm-secrets. This can be achieved by building a customized Docker image: more details can be found here.

Another option is to install SOPS or vals and helm-secret through an init container on the argocd-repo-server Deployment. This requires changing the initContainers args. More details can be found here.

However, both options have their drawbacks. Customizing the Docker image means maintaining an additional image while customizing initContainers commands results in a more complex values file for Argo CD, which can be challenging to maintain.

Is there a better or alternative way to manage Helm secrets? Let's continue exploring.

External Secrets Operator

A Quick Introduction to External Secrets Operator

The External Secrets Operator is a K8s operator that facilitates the integration of external secret management systems, such as AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, and Azure Key Vault, with K8s.

Simply put, this operator automatically retrieves secrets from these secrets managers using external APIs and injects them into Kubernetes Secrets.

Unlike helm-secrets which either stores encrypted data in the values file using another tool (SOPS) or refers to secrets stored in cloud secrets managers in the values file, the External Secrets Operator does not require including secrets.yaml as part of the Helm templates. It uses another custom resource ExternalSecret, which contains the reference to cloud secrets managers.

What does the custom resource do? Let's dive deeper to take a look under the hood of External Secrets Operator.

How External Secrets Operator Works

Here's an overview of how the External Secrets Operator works:

SecretStore configuration: First, we define a SecretStore resource that specifies the connection details and authentication credentials for the external secret management system with which we want to integrate.
ExternalSecret Configuration: Next, we create an ExternalSecret resource that defines the mapping between the external secrets and Kubernetes Secrets.
Syncing secrets: The External Secrets Operator continuously monitors the ExternalSecret resources. When a new or updated ExternalSecret is detected, the operator retrieves the specified secrets from the external secret management system using the configured SecretStore.
Automatic Synchronization: The External Secrets Operator periodically synchronizes the secrets based on a defined refresh interval.

External-Secrets Helm Chart Example

Let's see an example of using external secrets to manage Helm secrets.

The idea is simple: we do not include K8s Secrets as part of the Helm chart templates, but rather, we use ExternalSecret, which contains no sensitive information at all.

First, let's install the External Secret Operator itself:

We need to make sure access to AWS Secrets Manager is granted to the external secret operator. For a quick test with AWS Secrets Manager, we can create a secret containing our AWS credentials (do not do this in production) with access to Secrets Manager. Execute the following commands:

Make sure the access key has permission to access AWS Secrets Manager. For more information, check out AWS IAM policies.

This approach (access key as a K8s Secret) is only suitable for tutorials. For a production environment, it is recommended to use IAM roles for service accounts. Refer to the AWS official documentation and the External Secret Operator official documentation for more details.

Then, let's create a SecretStore pointing to AWS Secrets Manager in a specific account and region. Create a file named secretstore.yaml with the following content:

Apply the configuration by running the command:

Make sure to update the region with the appropriate AWS Secrets Manager region where your secrets are stored.

Then we can create an ExternalSecret as part of a Helm chart template that synchronizes a secret from AWS Secrets Manager as a Kubernetes Secret. Create a file named your-chart/templates/externalsecret.yaml with the following content:

The above ExternalSecret will create a K8s Secret named "helloworld" (specified in the "target" section) with one key "password", whose value is retrieved from MyTestSecret1.password in the AWS Secrets Manager.

Without changing anything else, we can simply install the chart by running:

After installation, we can verify the Secret is already created:

In the output, you should see the details of the created K8s Secret, which is automatically synchronized by the External Secrets Operator, fetching values from AWS Secrets Manager.

We can now utilize envFrom and secretRef in the Helm chart's Deployment to pass these secret values as environment variables.

Seamless Integration of External Secrets Operator with Continuous Deployment Tools

Unlike helm-secrets, which necessitates the installation of plugins to Helm and additional command-line tools like SOPS, the External Secrets Operator offers a more streamlined approach. It does not require any modifications to Helm or local binaries. Instead, it is solely installed on the K8s cluster side, utilizing an operator that needs to be installed and a SecretStore custom resource that must be defined.

Due to this inherent simplicity, integrating the external secret operator with continuous deployment tools such as Argo CD is effortless and trouble-free. No changes need to be made on the continuous deployment tool side. The only modification required is to the Helm chart itself: Secrets should not be included in the Helm chart template; instead, ExternalSecret should be used in the templates.

Vault Secrets Operator for Kubernetes Secrets and Helm Secrets

There is another major choice regarding managing secrets for Helm charts: the Vault Secrets Operator.

The Vault Secrets Operator works more or less similarly to the External Secrets Operator, but since it's made by HashiCorp, it only works with HashiCorp Vault. It works by watching for changes in the vault and synchronizing from the vault to a K8s Secret. The operator writes the source secret data directly to the destination Kubernetes Secret, ensuring that any changes made to the source are replicated to the destination over its lifetime.

It's worth noting that the External Secrets Operator also works with HashiCorp Vault. Still, if you are already using HashiCorp Vault, maybe the vault secrets operator can be a better choice since it's specifically designed for HashiCorp Vault and provides additional features tailored to Vault's capabilities, like dynamic secrets.

How to Automatically Restart Pods When Secrets Are Updated

In this section, we will discuss how to automatically restart pods when secrets are updated. This is an important requirement for most teams and companies. We will explore different approaches to achieve this.

If you are using a standard Helm chart without helm-secrets or the External Secrets Operator, you can use a hack to ensure that a Deployment's annotation section is updated when the secrets.yaml file changes. This can be done by using the sha256sum function. Here's an example:

This would work, and based on my experience, a lot of teams and companies use this in production, but in my eyes, this isn't ideal. For starters, it looks a bit messy in the annotations section.

In the case of helm-secrets, if the values are updated, we must do another helm secrets upgrade, since the values are either part of the Helm chart or injected in run time. This could be a little bit redundant, because what if neither the app nor the chart is updated, and only a secret value is updated? A full-on Helm upgrade seems a bit much.

Using the External Secrets Operator with Helm is even trickier: there is no secrets.yaml anymore, and the ExternalSecret only refers to secrets managers in the cloud, meaning the externalsecret.yaml doesn't change even if the values are updated in secrets managers, so we can't even use the checksum function.

To address these challenges, we recommend using Reloader. Reloader can watch for changes in secrets and ConfigMaps, and perform rolling upgrades on pods associated with DeploymentConfigs, Deployments, Daemonsets, Statefulsets, and Rollouts.

To install Reloader, simply run:

Once Reloader is installed, you can configure it to automatically discover Deployments where specific config maps or secrets are used. To do this, add the reloader.stakater.com/auto annotation to the main metadata of your Deployment as part of the Helm chart template:

This will discover Deployments automatically where a configmap or a secret is being used, and it will perform rolling upgrades on related pods when either is updated. Combined with External Secrets Operator, everything is solved without untidy hacks like the checksum annotation!

For more detailed usage of Reloader, check out the official doc here.

Summary

Learning to use helm-secrets can be challenging due to its integration with SOPS and the various encryption options it offers. Additionally, integrating helm-secrets with CD tools can result in increased operational overhead. However, if you only need to securely store sensitive data in the values file and do not plan on using a cloud secrets manager, helm-secrets is a suitable choice.

The Vault Secrets Operator and External Secrets Operator function similarly, but the Vault Secrets Operator, designed specifically for HashiCorp Vault, offers additional features such as dynamic secret generation.

For most users, the External Secrets Operator is likely the better choice as it is compatible with major public cloud providers' secrets managers and seamlessly integrates with CD tools and the Reloader tool. This conclusion is supported by the higher number of GitHub stars for the External Secrets Operator (3k stars) compared to the Vault Secrets Operator (0.3k stars) and helm-secrets (1k stars).

Of course, your own experience may differ, depending on your preferences. To make things easier, we created a workflow helping you choose among these solutions:

Secret in Helm: what's the best solution?

F.A.Q.

How does Helm Handle Secrets?

When it comes to handling secrets, Helm treats them as part of its templates, just like other Kubernetes resources such as Deployments and ConfigMaps. These templates can reference values defined in the values.yaml file. However, since secrets contain sensitive information, it is bad practice to store clear-text credentials directly in the Helm values file. Instead, the values file should only contain sample default values.

How Do You Use Secrets with Helm Charts?

There are two options for using Secrets with Helm Charts. The first is to encrypt the sensitive information in the Helm values file. The second option is to use an alternative mechanism to handle Kubernetes Secrets, rather than including them as part of the Helm chart.

How Does helm-secrets Work?

helm-secrets provides a solution for encrypting sensitive values files using SOPS. This ensures that the encrypted files can be safely committed to git repositories and shared among team members. SOPS also offers access control capabilities and supports different encryption methods.

Additionally, helm-secrets provides the option to store sensitive data in cloud secrets managers instead of the values file. These values can then be injected during helm installs.

What is the Alternative to helm-secrets?

For those looking for an alternative to helm-secrets, the External Secrets Operator is a reliable choice. HashiCorp Vault users may also consider the Vault Secrets Operator.

Handling Secrets with AWS Secrets Manager

Tiexin Guo — Sat, 07 Oct 2023 03:33:17 +0000

In my previous tutorials, we looked at Azure Key Vault and Google Secret Manager:

How to Handle Secrets with Azure Key Vault: In this piece, we had a look at the Zero Trust security strategy, how to put it into practice to secure applications and data, and how secrets managers can help to achieve the Zero Trust goal. We also included a tutorial on Kubernetes/SPC to use secrets from secret managers.
How to Handle Secrets with Google Secret Manager: In this piece, we did a tutorial on using secrets from secret managers in your CI workflows (GitHub Actions).

If you haven't read them yet, please give them a quick read, because even if you are not an Azure or a GCP user, they still might be worth reading.

Today, we will have a look at AWS's equivalent: AWS Secrets Manager. We will do a quick introduction and some tutorials (cheat sheet included).

Let's get started.

1 What is AWS Secrets Manager?

In general, a secret manager helps you improve your security posture by getting rid of hard-coded credentials in application source code, CI/CD workflows, and maybe even Kubernetes YAML manifests. Hard-coded credentials are replaced with a runtime call to the secret manager (or other mechanisms) to retrieve/sync credentials dynamically when you need them, and this helps avoid possible compromise by anyone who can inspect your application or the components.

AWS Secrets Manager is AWS's offering, which can help you manage, retrieve, and even rotate secrets, which can be database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles.

2 What Secrets Can Be Stored in Secrets Manager?

You can manage secrets such as database credentials, on-premises resource credentials, SaaS application credentials, third-party API keys, and Secure Shell (SSH) keys.

Please note that although Secrets Manager enables you to store a JSON document which allows you to manage any text blurb that is 64 KB or smaller, making it possible to be used in a wide range of scenarios, for certain types of secrets, there are better ways to manage them in AWS, for example:

AWS credentials: we can use AWS IAM instead of storing/accessing AWS credentials with Secrets Manager.Management.
Encryption keys: use AWS KMS service.
SSH keys: use AWS EC2 Instance Connect instead.
Private keys and certificates: use AWS Certificate Manager.

3 Secrets Manager Features

AWS Secrets Manager enables us to store, retrieve, control access to, rotate, audit, and monitor secrets centrally. Here are some features worth mentioning:

Fine-grained access control policies: IAM policies and resource-based policies can be used together to manage access to our secrets.
Multi-region replication: we can automatically replicate secrets to multiple AWS Regions to meet our disaster recovery and cross-regional redundancy requirements.
Secrets rotation: the credentials for AWS RDS (Relational Database Service), DocumentDB, and Redshift can be natively rotated by Secrets Manager. We can also extend Secrets Manager to rotate other secrets, such as credentials for Oracle databases hosted on EC2 or OAuth refresh tokens, by modifying sample AWS Lambda functions available in the Secrets Manager documentation.
Audit and monitoring: Secrets Manager can be integrated with many AWS logging, monitoring, and notification services, making auditing and monitoring secrets easy. For example, after enabling CloudTrail for a region, we can audit when a secret is created or rotated by viewing CloudTrail logs; for another, we can configure CloudWatch to receive email messages using SNS when secrets remain unused for a period or configure CloudWatch events to receive push notifications when Secrets Manager rotates our secrets.

4 How Secrets Manager Works under the Hood

To understand how it works, think of Secrets Manager as a super secure vault for our sensitive data. It uses powerful encryption keys, which we own and store in AWS Key Management Service (KMS). We are in full control of who can access our secrets with the help of AWS Identity and Access Management (IAM) policies.

When a secret needs to be retrieved, Secrets Manager ensures it's done securely. It first decrypts the secret, then sends it to our application over a secure connection (TLS). And don't worry, it doesn't write down or stash our secret anywhere else.

For the encryption part, Secrets Manager uses what's called envelope encryption (AES-256 encryption algorithm) to encrypt our secrets in AWS Key Management Service (KMS):

When start using Secrets Manager, the AWS KMS keys can be specified to encrypt secrets. If not specified, don't worry, because Secrets Manager will automatically create AWS KMS default keys.
When a secret is stored, Secrets Manager works some magic behind the scenes. It asks KMS for plain text and an encrypted data key. Then, it uses that plain text data key to encrypt secrets right in memory. This encrypted secret and data key are then stored safely away.
When accessing a secret, Secrets Manager decrypts the data key (using AWS KMS default keys) and uses the plain text data key to decrypt the secret. Rest assured, the data key is always stored encrypted and never written to the disk in plain text. Plus, the secret won't be written or cached to persistent storage in plain text either.

5 Creating a Secret in Secrets Manager

5.1 Create/Update

We can create a secret in the AWS web console:

In this example, I chose "other type of secret", and named it "MyTestSecret1":

Other settings are left as default.

Of course, as a hardcore DevOps engineer, I prefer the CLI way because the web console is too user-friendly and I don't like it (just joking).

To create a secret using the CLI:

aws secretsmanager create-secret \
  --name MyTestSecret2 \
  --description "My test secret created with the CLI." \
  --secret-string "{'user':'tiexin','password':'EXAMPLE-PASSWORD'}"

The command above creates a secret with two key-value pairs.

Alternatively, we can store the content of the secret in a JSON file, and create a secret using that JSON file. Create a file named mycreds.json with the following content:

{
  "username": "tiexin",
  "password": "EXAMPLE-PASSWORD"
}

And create a secret from the file:

aws secretsmanager create-secret \
  --name MyTestSecret3 \
  --secret-string file://mycreds.json

We can get the value with the get-secret-value command:

aws secretsmanager get-secret-value --secret-id MyTestSecret3

And we can also update the value using the put-secret-value command:

aws secretsmanager put-secret-value \
  --secret-id MyTestSecret3 \
  --secret-string "{'user':'tiexin','password':'NEW-EXAMPLE-PASSWORD'}"

The above command creates a new version (discussed in the next subsection) of a secret with two key-value pairs.

5.2 Secret Versions

A secret has versions that hold copies of the encrypted secret value. When you change the secret value or the secret is rotated, Secrets Manager creates a new version. Secrets Manager doesn't store a linear history of secrets with versions. Instead, it keeps track of three specific versions by labeling them, like AWSCURRENT, AWSPREVIOUS. A secret always has a version labeled AWSCURRENT, and Secrets Manager returns that version by default when you retrieve the secret value.

For example, if we have a look at the secret "MyTestSecret3", on which we did an update:

aws secretsmanager list-secret-version-ids --secret-id MyTestSecret3
{
    "Versions": [
        {
            "VersionId": "a11a8133-96ae-4abc-9bfb-e737ae39266e",
            "VersionStages": [
                "AWSPREVIOUS"
            ],
            "CreatedDate": 1692428755.262,
            "KmsKeyIds": [
                "DefaultEncryptionKey"
            ]
        },
        {
            "VersionId": "a2477f83-02a9-457c-b473-c9589c5d7309",
            "VersionStages": [
                "AWSCURRENT"
            ],
            "CreatedDate": 1692428770.661,
            "KmsKeyIds": [
                "DefaultEncryptionKey"
            ]
        }
    ],
    "ARN": "arn:aws:secretsmanager:ap-southeast-1:xxx:secret:MyTestSecret3-vGMlXZ",
    "Name": "MyTestSecret3"
}

We can see that there are two versions of it.

5.3 List/Find

We can use the command aws secretsmanager list-secrets to list all secrets, and use the --filter parameter to filter wanted secrets:

aws secretsmanager list-secrets --filter Key="name",Values="MyTest"

This will show all secrets whose names start with "MyTest".

Other than the "name" filter we used in the above command, there are other filter keys you can use:

description
tag-key
tag-value
owning-service
primary-region
all (searches all of the above keys)

5.4 Delete

Because of the critical nature of secrets, AWS Secrets Manager intentionally makes deleting a secret difficult.

Secrets Manager DOES NOT immediately delete secrets by default. Instead, Secrets Manager immediately makes the secrets inaccessible and scheduled for deletion after a recovery window of a minimum of 7 days. Until the recovery window ends, you can recover a secret you previously deleted.

To delete a secret:

aws secretsmanager delete-secret \
  --secret-id MyTestSecret3 \
  --recovery-window-in-days 7

During the recovery window, if wanted, we can recover the secret by running:

aws secretsmanager restore-secret \
  -secret-id MyTestSecret3

With the CLI, it's possible to do a force-delete without a recovery window but please use it with caution:

aws secretsmanager delete-secret \
  --secret-id MyTestSecret2 \
  --force-delete-without-recovery

6 Accessing Secrets

In the previous two tutorials (link here and here), we have covered a few ways to access secrets stored in a secret manager, like from an application directly using the SDK provided by the public cloud provider; from CI pipelines, like GitHub Actions; and from a Kubernetes cluster using the Secret Provider Class (SPC). Although the previous two tutorials are done with Azure and GCP, the principles are the same for AWS.

6.1 From Applications Directly Using SDK

For example, to access secrets from a Python application, we will need:

Python 3.6 or later.
botocore 1.12 or higher. See AWS SDK for Python and Botocore.
setuptools_scm 3.2 or higher. See https://pypi.org/project/setuptools-scm/.

To install the component, use the following command.

$ pip install aws-secretsmanager-caching

The Python application needs the following IAM permissions:

secretsmanager:DescribeSecret
secretsmanager:GetSecretValue

For example, if you are running the Python app on an EC2 virtual machine, the instace profile of that EC2 needs the above permissions.

The following code snippet example shows how to get the secret value for a secret named MyTestSecret1:

import botocore
import botocore.session
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig

client = botocore.session.get_session().create_client('secretsmanager')
cache_config = SecretCacheConfig()
cache = SecretCache( config = cache_config, client = client)

secret = cache.get_secret_string('MyTestSecret1')
print(secret)

For more detail on accessing secrets in a Python app using the SDK, see the official doc here.

6.2 From CI Workflows (Example: GitHub Actions)

Similarly, as we did in the GCP tutorial, we can use a secret in a GitHub job to retrieve secrets from AWS Secrets Manager and add them as masked Environment variables in our GitHub workflows.

To do this, we first need to allow GitHub Actions to access AWS Secrets Manager, which can be achieved by using the GitHub OIDC provider. My previous blog here details how to do this, allowing us to use short-lived credentials and avoid storing additional access keys outside of Secrets Manager.

The IAM role assumed by the GitHub Actions must have the following permissions:

GetSecretValue on the secrets we want to retrieve
ListSecrets on all secrets.

Then we can simply add a step in our GitHub Actions workflow using the following syntax to access secrets from Secrets Manager:

- name: Step name
  uses: aws-actions/aws-secretsmanager-get-secrets@v1
  with:
    secret-ids: MyTestSecret1
    parse-json-secrets: (Optional) true|false

6.3 From K8s/EKS Clusters with the Secret Provider Class (SPC)

Similarly, as we did in the Azure tutorial, we can use Kubernetes Secret Provider Class to access secrets from Kubernetes clusters.

We use YAML to describe which secrets to mount in AWS EKS. The SPC is in the following format:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: <NAME>
spec:
  provider: aws
  parameters:
    region:
    failoverRegion:
    pathTranslation:
    objects:

Here's an example:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "MySecret"
        objectType: "secretsmanager"

For more details on using SPC with AWS EKS and Secrets Manager, see the official doc here.

7 Tutorial: AWS EKS + External Secrets Operator + Secrets Manager

Today, let's do a tutorial using a different approach for accessing Secrets Manager from Kubernetes clusters: the External Secrets Operator.

7.1 The Secret Store CSI Driver/SPC Method Explained

SPC works by ways of Secrets Store CSI Driver to mount secrets, and since the CSI is Container Storage Interface, it mounds secrets as a container storage volume. This means secrets must be mounted as a volume on the Pod.

In the cloud-native era, following the 12-factor app principle, we don't want to write our apps in a way that they read secrets from a local file path. We prefer to read secrets from ENV vars, for example, using a Kubernetes Secret as ENV vars with envFrom and secretRef. In the case of the Secrets Store CSI Driver/SPC method, it can sync secrets from a secret manager as a Kubernetes Secret, but the volume is still required to do so, which seems to be a bit of redundancy since we won't be using that volume anyways.

7.2 External Secret Operator

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager (and HashiCorp Vault, Google Secrets Manager, Azure Key Vault, etc.) The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

Although it sounds very similar to the SPC method, the key difference is that the External Secret Operator does not use the CSI driver, hence requiring no volumes to sync secrets from a secret manager.

With the key difference cleared out of the way, let's continue with the tutorial.

7.3 Create an EKS Cluster

On macOS, the easiest way to install eksctl is by using brew:

brew tap weaveworks/tap
brew install weaveworks/tap/eksctl

For other operating systems and installation methods, see the README here.

After installation, we can run a simple command to bootstrap a cluster:

eksctl create cluster

This command will create an EKS cluster in the default region (as specified by AWS CLI config) with one managed node group containing two m5.large nodes.

7.4 Install External Secret Operator

After the cluster is up and running, we can install the External Secret Operator using helm:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

Then, we need to allow External Secret Operator to access AWS Secrets Manager. Since this is a tutorial and not meant for a production environment, we will do so by creating a secret containing our AWS credentials which has access to Secret Manager:

echo -n 'KEYID' > ./access-key
echo -n 'SECRETKEY' > ./secret-access-key
kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key

Please note that this is only for tutorial purposes; for a production environment, use the IAM role for service accounts. See the AWS official doc here and External Secret Operator official doc here.

7.5 Create a Secret Store

A SecretStore points to AWS Secrets Manager in a certain account within a defined region. Create a file named secretstore.yaml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

And apply it by running: kubectl apply -f secretstore.yaml. Remember to update the region to the region where you store your secrets in AWS Secrets Manager.

7.6 Create an External Secret

Then, we create an External Secret that syncs a secret from AWS Secrets Manager as a Kubernetes Secret. Create a file named externalsecret.yaml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: secret-to-be-created
    creationPolicy: Owner
  data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: MyTestSecret1
  data:
  - secretKey: username
    remoteRef:
      key: MyTestSecret1
      property: user
  - secretKey: password
    remoteRef:
      key: MyTestSecret1
      property: password

And apply it by running: kubectl apply -f externalsecret.yaml.

The ExternalSecret above means:

Create a K8s Secret named "secret-to-be-created" (defined in the "target) section
The K8s Secret will have two keys, one is "username" (from MyTestSecret1.user in AWS Secrets Manager) and the other is "password" (from MyTestSecret1.password in AWS Secrets Manager).

After applying the YAML, we can verify by running:

$ kubectl describe secret secret-to-be-created
Name:         secret-to-be-created
Namespace:    default
Labels:       reconcile.external-secrets.io/created-by=default_example
Annotations:  reconcile.external-secrets.io/data-hash: 49d444f36fb63e85d07ccc7a8d10441c

Type:  Opaque

Data
====
password:  4 bytes
username:  6 bytes

We can see that the Kubernetes secret is already created automatically by the External Secrets Operator, synchronizing values from AWS Secrets Manager. From here, you can use envFrom and secretRef in your Kubernetes Deployment/Pod.

8 Tear Down

Delete all secrets created in this tutorial:

aws secretsmanager delete-secret \
  --secret-id MyTestSecret1 \
  --force-delete-without-recovery

aws secretsmanager delete-secret \
  --secret-id MyTestSecret2 \
  --force-delete-without-recovery

aws secretsmanager delete-secret \
  --secret-id MyTestSecret3 \
  --force-delete-without-recovery

Delete the EKS cluster:

eksctl delete cluster --name NAME_OF_THE_EKS_CLUSTER_CREATED

9 Cheat Sheet

9.1 Secrets CRUD

Create:

aws secretsmanager create-secret --name MyTestSecret \
  --description "xxx" \
  --secret-string "xxx"

Create using a file:

aws secretsmanager create-secret \
  --name MyTestSecret \
  --secret-string file://mycreds.json

Read:

aws secretsmanager get-secret-value --secret-id MyTestSecret

Update:

aws secretsmanager put-secret-value \
    --secret-id MyTestSecret \
    --secret-string "yyy"

List versions:

aws secretsmanager list-secret-version-ids --secret-id MyTestSecret

List/search:

aws secretsmanager list-secrets --filter Key="name",Values="MyTest"

Delete:

aws secretsmanager delete-secret \
  --secret-id MyTestSecret \
  --recovery-window-in-days 7

Force delete without a recovery window:

aws secretsmanager delete-secret \
  --secret-id MyTestSecret \
  --force-delete-without-recovery

9.2 Access Secrets from GitHub Actions

OIDC provider usage example: https://blog.gitguardian.com/securing-your-ci-cd-an-oidc-tutorial/

GitHub Actions step to read a secret:

- name: Step name
  uses: aws-actions/aws-secretsmanager-get-secrets@v1
  with:
    secret-ids: MyTestSecret1
    parse-json-secrets: (Optional) true|false

9.3 SPC/External Secrets Operator

SPC format reference:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "MySecret"
        objectType: "secretsmanager"

SecretStore format reference:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

ExternalSecret template reference:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: secret-to-be-created
    creationPolicy: Owner
  data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: MyTestSecret1
  data:
  - secretKey: username
    remoteRef:
      key: MyTestSecret1
      property: user
  - secretKey: password
    remoteRef:
      key: MyTestSecret1
      property: password

Summary

In this article, we did a quick introduction to AWS Secrets Manager and demonstrated how to use AWS CLI to do CRUD for secrets. Then we covered how to use secrets using SDK, in CI workflows, and in Kubernetes with SPC.

Due to space limitations, in this tutorial, we didn't rewrite much on the SDK/SPC/GitHub Actions usage of secrets. If you have more questions on these parts, please refer to the previous two tutorials on Azure and GCP secret managers, where more details are available.

Then We did an extra extended tutorial on the External Secrets Operator; choose between that and the SPC method based on your requirements.

In the end, like always, a detailed cheat sheet is provided, covering the most common use cases regarding AWS Secrets Manager.

I hope this tutorial and the whole series are useful to AWS/GCP/Azure users. Please like, comment, subscribe. Thanks!

Platform Engineering: Building Your Developer Portal with Backstage (Pt 1)

Tiexin Guo — Sun, 09 Jul 2023 03:02:47 +0000

Platform Engineering: Building Your Developer Portal with Backstage - Part 1

In my previous article, we looked at platform engineering: what it is, how it became a thing, platform engineering V.S. DevOps, and how it could help improve security.

If you haven't read it yet, here is the link to it for you:

Platform Engineering and Security: A Very Short Introduction

Today, let's get hands-on and build a developer portal from scratch.

1 What Are We Talking about When We Talk about Developer Portal

Since platform engineering is all about self-service, the internal developer portal is crucial in the new paradigm, in which all the self-service capabilities and integrations are implemented.

Given the significance of the developer portal, before building one, we need to figure out what exactly we are expecting from it. Typically, to get the most out of platform engineering, a developer portal would contain some key features, like:

a service catalog: giving developers a bird's-eye view of all the projects, services, deployment status, documentation, ownership, and even on-call schedules and incidents, etc.;
some kind of repository scaffolding/project generation tool: for example, something like cookiecutter, which enables developers to spin up new services from within the portal itself;
customization: you might want to integrate whatever toolchain you use into your developer portal to make it a genuinely unified one-stop experience. Possible integrations include Kubernetes, CI/CD status, on-call schedules, incident management systems, secrets managers, etc.

At this point, the portal can handle everything, like getting information about services, on-call schedules, triggering incidents, deploying things, and dealing with incident management. With these features, developers can have a truly unified experience where creating services and subsequently keeping track of and operating/maintaining those services in one place, maximizing the potential of a developer portal.

2 Choosing the Tool: Backstage

Now that we know what we really want from a developer portal, let's build one.

Since it will have multiple features, the designing/coding part could be a massive project in itself, and not all teams could afford that. That said, we need a tool for building the portal for a quick start, and luckily, we've got one - Backstage.

Backstage is an open platform for building developer portals. Backstage itself isn't a developer portal but a tool to build your developer portal. Think of "create-react-app" V.S., the actual react app you created with it. Out of the box, Backstage includes:

software catalog for managing all your software, such as microservices, libraries, data pipelines, websites, and ML models;
software templates for quickly spinning up new projects and standardizing your tooling with your organization's best practices;
docs for making it easy to create, maintain, find, and use technical documentation, using a "docs like code" approach;
a growing ecosystem of opensource plugins that further expand Backstage's customizability and functionality.

It can be so flexible in its architecture: it has a frontend written in React/TypeScript and a backend in Node.js, and it extends its power by adding plugins (which we will probably cover in the second part of this tutorial.

What's more, Backstage (created by Spotify) is now hosted by the Cloud Native Computing Foundation (CNCF) as an Incubation level project, meaning you can get all the community support you want. It's also got office hours, where you can join interactively to learn precisely how the opensource platform can drive better developer effectiveness and experience every Thursday.

Enough said; today, we will start from scratch and build a developer portal ourselves. After this tutorial, you can bootstrap a new service using templates with baked-in security CI workflows, check the CI status, and view the documentation for that service in the same place.

Without further adieu, let's get started.

3 Building The Portal

3.1 Prerequisites

Not much; probably all DevOps engineers already got them:

A Unix-based operating system. For example, you can run this on macOS, Linux, or Windows Subsystem for Linux (WSL).
curl, git, Docker
Node.js, yarn

3.2 Creating the Portal

Run:

npx @backstage/create-app@0.5.2-next.3

Author's note:

The reason we use backstage/create-app@0.5.2-next.3 instead of backstage/create-app@latest is because we need this feature to create secrets in GitHub to be used in GitHub Actions workflows. This feature has been developed, but as of now (June 18, 2023), it's not yet integrated into the latest version of backstage/create-app.

If you read this article later when the latest version points to a version released in June 2023 (or later), you can safely run npx @backstage/create-app@latest instead without specifying the weird version.

This command gives you an interactive mode where you need to enter the name, for example, "my-portal", press enter, then wait till the app is finished.

Enter the directory, and run:

yarn dev

And you should have the portal up and running already! It's that simple! Now, you can poke around with it for a bit and have a feeling of the software catalog, templates, documents, and stuff there:

But, at this moment, it's not of much use yet. So, let's continue configuring it.

4 GitHub Authentication and Integration

Since the developer portal will be in charge of bootstrapping new repositories, it requires permissions to operate GitHub, and that's why we need to do a GitHub authentication and integration.

4.1 Personal Access Token for Integration

While using GitHub Apps might be the best way to set up integrations because of its fine-grained permission settings, for this tutorial, we'll use a Personal Access Token for a quicker start:

Create your Personal Access Token by opening the GitHub token creation page.
Use a name to identify this token and put it in the notes field. Choose the number of days for expiration.
If you have a hard time picking a number, we suggest going for 7 days; it's a lucky number :) (And we will only be testing it locally, not running in production.)
For this tutorial, let's set the scope to the maximum so that your experience won't be blocked by struggling with GitHub permissions. Although, please note that you should NEVER do this in production!

Then, export the token as an environment variable:

export GITHUB_TOKEN=xxx

In the app-config.yaml file, change the integrations section to the following:

integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN} # leave it like this, read values from env var

4.2 Creating a GitHub OAuth App

Go to https://github.com/settings/applications/new to create your OAuth App. The Homepage URL should point to Backstage's frontend; in our tutorial, it would be http://localhost:3000. The Authorization callback URL will point to the auth backend, which will most likely be http://localhost:7007/api/auth/github/handler/frame.

Then, let's open our app-config.yaml file again and configure the authentication by updating the auth section to the following:

auth:
  environment: development
  providers:
    github:
      development:
        clientId: YOUR CLIENT ID # put your values here
        clientSecret: YOUR CLIENT SECRET # put your values here

Please note that storing the client secret in the config file as a hardcoded secret is against security best practices and should only be used for local development. For production usage, we can read them from environment variables, following the 12-factor app rules

After these config changes, we must stop our yarn dev servers and re-run yarn dev.

5 Creating a Template

Next, let's prepare a software template, which could be used to bootstrap a new service in no time.

The template should contain the following:

some basic source code/directory structure common to your services
some documentation
some CI workflows to test/build/deploy your service

For this tutorial, we will use this template that I created as an example.

The directory structure is relatively simple; there are only two parts:

a skeleton folder,
a template.yaml file.

5.1 The `skeleton` Folder

The skeleton folder contains all the templates that will be rendered when using this template to create a new service, and the variables are in the format of ${{ values.varName }}.

If you are familiar with Helm, YAML, or Go templates (or just about any template tool), you will find this easy to read and understand at no cost whatsoever.

The only item worth mentioning is a file named catalog-info.yaml, which is used by Backstage, so this file must exist; otherwise, you can't register the created service as a component in the portal.

As you can see, in the template, we have already got some baked-in GitHub Actions workflows, one would test on pull requests and push to the main branch, and the other will use ggshield to scan repositories for hardcoded secrets.

In this way, we can put all the CI/CD best practices in the template so that when others launch a new service, they already have everything they need with security features enabled by default.

5.2 The `template.yaml` File

The template.yaml file defines how the template looks in the portal UI and what it actually does.

It is long and seems overwhelming at first glance, but once you have a closer look at it, you will notice it's pretty straightforward:

the syntax is like a Kubernetes custom resource;
it has two major parts, one is parameters, and the other is steps;
the parameters define required input and their types when using this template;
the steps define what actually happens when you execute this template, and it looks a whole lot just like a GitHub Actions workflow.

Parameters example:

parameters:
  - title: Provide some simple information
    required:
      - service_name
      - owner
    properties:
      service_name:
        title: Name
        type: string
        description: Unique name of the service.
        ui:field: EntityNamePicker
      description:
        title: Description
        type: string
        description: Help others understand what this service is for; optional.
      owner:
        title: Owner
        type: string
        description: Owner of the component
        ui:field: OwnerPicker
        ui:options:
          allowedKinds:
            - Group
  ...

Steps example:

steps:

- id: template
  name: Fetch Skeleton + Template
  action: fetch:template
  input:
  url: ./skeleton
  copyWithoutTemplating: - .github/workflows/\*
  values:
  serviceName: ${{ parameters.service_name }}
  description: ${{ parameters.description }}
  destination: ${{ parameters.repoUrl | parseRepoUrl }}
  owner: ${{ parameters.owner }}
  ...

From the above file, we can infer what exactly it defines:

first, you need to enter two input parameters: service_name and owner;
then you need to choose a repository location with one extra parameter (GITGUARDIAN_API_KEY, to be used in CI pipelines);
it then fetches the template, renders it, publishes it to GitHub, and registers it in our portal.

5.3 Register the Template

Finally, let's add our template to our portal's catalog so that others can use this template.

The most straightforward configuration for the catalog is to declaratively add locations pointing to YAML files with static location configuration. Locations are added to the catalog under the catalog.locations key in the app-config.yaml file:

catalog:
  locations:
    - type: url
      target: https://github.com/IronCore864/backstage-test-template/blob/main/template.yaml
      rules:
        - allow: [Template]

The rule above allows us to add a template from the specified URL.

Remember to restart yarn dev servers.

6 Putting Everything Together

Now that we've got everything ready, it's time to see the magic in action.

Visit http://localhost:3000, and click the "Create" button, choose our template:

Input necessary information. For the GitGuardian API key, create one here: https://dashboard.gitguardian.com/api/personal-access-tokens.

When everything is set, click "next step", and the magic will happen:

Everything should be created now.

We can view it in our catalog:

And we've also got documentation created and rendered:

Last but not least, let's check out the CI status:

It seems the pipelines are already finished successfully, and we can click on them for more details with detailed steps and logs.

If you prefer to view them in the CI software (in this case, GitHub Actions), you can click the corresponding links to jump to them.

For your reference, this repo is the one I created using the template above.

This means no matter what toolchain the team uses, the team members don't have to remember 10 different URLs for 10 different tools, and they don't have to keep those 10 tabs open all the time. When they need some information, any information, they go to the developer portal, and they've got everything, and that's precisely the power of an internal developer portal.

7 Summary

In the first part of the tutorial, we reviewed the features of developer portals, learned how to use the opensource Backstage tool to create a portal ourselves, configured our portal with GitHub, created a software template, and bootstrapped a service from it.

A developer portal could be much more than this if more and more integrations are added. Imagine if you deploy services in Kubernetes, use Argo CD for GitOps deployment, use HashiCorp Vault for secrets management, and all the integrations are in your portal: when you need to check the deployment status, when you want to see the actual resources in K8s, you don't have to visit Vault, Argo CD, K8s Dashboard. Heck, you don't even have to remember the URLs for them or even know the existence of those things because you have the one tool to rule them all, and that's the developer portal.

This tutorial only works for a local quick start; there are many things to think about for production usage. For example, we use static config for now; nothing is persistent; if you restart the dev server, the catalog info is lost. For this, we need to configure Postgres for our portal. For another example, we use yarn dev to start both the frontend and backend; for production usage, you might want to separate the frontend from the backend, deploy them in K8s as containers, and maybe create Ingress and stuff for them.

In the next part of this tutorial, we will look at the Backstage plugin's mechanism to see how we can expand its power to the next level.

If you enjoy this article, please like, comment, and subscribe. See you in the next part!

Platform Engineering and Security: A Very Short Introduction

Tiexin Guo — Tue, 30 May 2023 06:25:51 +0000

I'm a DevOps engineer. I hope that's still a thing because, according to some, "DevOps is dead; long live platform engineering."

There is no denying that recently we started to see terms like "platform engineering," "developer portal," or even "Backstage" a lot more often than before.

This article will answer a few essential questions: is DevOps really dead? What is platform engineering? What are the differences? How does platform engineering handle DevSecOps/security?

Is DevOps Dead?

Short answer, no.

If I learned anything over the years in the tech world, it's the fact that technologies, frameworks, and methodologies never really die. They grow, they adapt, and they evolve.

For example, think agile development. What's the first time you heard of that? Is it dead now? That's the point.

But, in a way, DevOps did "die." Although it didn't die in the conventional sense, where it simply disappeared from the tech community, it did "die" in a way that many companies and teams trying to utilize DevOps failed to meet their expectations.

This is not news. Back in 2019, a data analyst from Gartner, a technological research and consulting firm, predicted that by 2022, 75% of DevOps initiatives would fail to meet expectations, and this number might increase to 90% in 2023.

It's worth pointing out that those DevOps initiatives failed not because of technological challenges but because of overlooked human factors like trust, ownership, and teamwork.

Initially, DevOps was touted as the ultimate solution to all issues within traditional operations silos, promising to revolutionize the industry. Still, many teams have unfortunately fallen short of their original expectations. Worse, it seems that, in the end, developers don't actually want to do Ops work!

Devs Don't Want to Do Ops?

To be fair, it depends on who you ask. Some developers think the DevOps "you build it, you run it" paradigm is definitely beneficial, even sometimes necessary; others don't want to touch operations at all.

Take, for example, this poll:

Devs… be real with me. Do y’all want to do Ops? 🧵

— Luca (@luca_cloud) August 24, 2022

The responses highlight the divide, with 41.8% of respondents saying yes to operations tasks, 42.1% saying no, and 16.1% being indifferent.

As it turns out, developers tend to steer clear of infrastructure and operations, which is not very positive for the success of DevOps.

The consequence is that platform engineering has become increasingly popular in response to this trend.

Platform Engineering

Platform engineering, just like DevOps, emerged in response to the increasing complexity of modern software architecture.

In today's world, non-expert end users, such as developers, are frequently tasked with operating a complex array of services that can be incredibly challenging to manage.

While DevOps tries to solve the collaboration and velocity problem of the software development lifecycle (SDLC) from a cultural standpoint—where shared responsibility, continuous growth, and learning mindset are valued, teamwork, best practices, and modern toolchains encouraged—platform engineering tries to approach the problem from a different angle: that of self-service.

Let's have a look at a concrete example:

Consider a scenario where a team of developers requires a relational database (RDS) for their new application, but they lack the necessary knowledge to create one in a specific cloud provider, such as AWS, using the appropriate automation tool, such as Terraform. Not all developers are familiar with the infrastructure and orchestration of their systems.

The DevOps approach to this problem involves creating a cross-functional team that includes an expert in AWS services and infrastructure as code who possesses the proper permissions to provision the cloud resources automagically by writing Terraform modules and deploying the RDS in AWS.

On the other hand, in the platform engineering approach, there should be an integrated product that serves as the "internal developer portal", so that the developers request an RDS in AWS through self-service, and the platform will automatically create it.

Note that this is only one example; internal developer portals can achieve way more if appropriately crafted.

In short, platform engineering involves designing and constructing an integrated product that includes toolchains and workflows to address the operational requirements of the entire SDLC. This approach enables developers to access the appropriate level of self-service and abstraction that suits their organization or team in the cloud-native era.

Platform Engineering vs DevOps

Although these self-service capabilities improve the developer experience and productivity, it's not for everyone, especially not for smallish teams/companies: even if you decide to purchase some internal developer portal-as-a-service instead of building it from scratch, you still need to maintain automation and templates behind the scene: think of the previous RDS-in-the-cloud example, where you need to have the knowledge to maintain Terraform templates to generate the module.

If a team or company is too small for platform engineering, DevOps may be a more effective approach. In smaller environments, collaboration tends to be more intimate, the pace is more agile, and there is typically less organizational friction.

It is important to note that platform engineering does not serve as a replacement for DevOps. It does not emerge to displace or replace DevOps. Although many say that DevOps is dead with the rise of platform engineering, it is not a situation where one will replace the other: the two approaches can come together.

Platform engineering can be seen as an evolution of DevOps, as it shares the same objective and can enhance the effectiveness of DevOps.

Likely, we will still see DevOps culture a lot, and it's also likely that many software engineering organizations will have established platform teams to help bring together software developers and IT operations.

Where Does Security Fit in this New Paradigm?

We can't talk about DevOps without talking about security (or DevSecOps); after all, security is job zero.

DevOps tackles security by "shifting left." In contrast to traditional software development methods, where security-related tasks were only addressed at the end of the SDLC, which could lead to project delays and was not scalable, DevOps/DevSecOps incorporates security at every stage of the SDLC as early as possible, using automation tools. This approach is known as "baking in" security.

To illustrate, when writing code, developers consider potential security concerns, such as securely storing and retrieving secrets and credentials, rather than waiting for a final audit or review before pushing to production. Additionally, when creating virtual machine or container images, vulnerability scans are automatically conducted during each build, and any vulnerabilities found are immediately addressed, rather than conducting a one-time scan before shipping to production.

This is "shift left."

All these methods of dealing with securities in a more agile, responsive, and automated "shifting-left" manner are still applicable to platform engineering; there is no contradiction here regarding DevOps and platform engineering.

With the assistance of platform engineering, internal developer portals can automate the entire process to a greater extent. Consider three concrete examples:

Firstly, imagine creating a new microservice. Instead of writing boilerplate code and manually assembling CI/CD pipelines, Dockerfiles, and Kubernetes manifests, the internal developer portal can bootstrap your entire repository using existing templates. These templates include scaffolding code for the app, as well as pre-configured pipelines that follow DevSecOps best practices. They also scan for hard-coded secrets in the source code and lint YAMLs, among other things.

Secondly, imagine needing to store secrets. Instead of searching for the secrets manager your team is using and determining whether you have permission to create secrets, you can go to the internal developer portal, which integrates with your secrets manager. Here, you can create secrets in a self-service manner without worrying about the underlying details, thanks to the appropriate level of abstraction provided by platform engineering.

Finally, imagine launching a new virtual machine in the cloud. Instead of writing automation code from scratch or copying and pasting from the existing codebase and reviewing for security groups settings before deploying, what if you can simply go to your internal developer portal, click a button, and everything is done using a template maintained and secured by the platform engineering team?

All these are the beauties of platform engineering:

- Self-service

- The right amount of abstraction

- Baked-in best practices and automation

Summary

I hope this introduction to platform engineering has sparked your interest! You should now have a better understanding of why it emerged and its core proposition value for developers, as well as the differences with DevOps and how security is addressed with this new paradigm. If you are interested in platform engineering, please subscribe, as we will be working on some tutorials to help you build your own developer portal from scratch.

Stay tuned!

Open Policy Agent with Kubernetes - Tutorial (Pt. 2)

Tiexin Guo — Wed, 22 Feb 2023 16:26:15 +0000

In my previous articles, we discussed what Policy-as-Code is, why we need it, and how to use the Open Policy Agent (OPA) tool. If you haven't read the introduction yet, please take some time to read it first:

What is Policy-as-Code? An Introduction to Open Policy Agent

Following the OPA introduc how to use OPA to enforce policies inside a Kubernetes cluster. Here's the link to the first part of the tutorial:

Open Policy Agent with Kubernetes - Tutorial (Pt. 1)

TL;DR: we used OPA as the admission controller with the kube-mgmt sidecar enforcing ConfigMap-based policies.

Today, we will look at another way of using OPA in a Kubernetes cluster: Gatekeeper - policy controller for Kubernetes.

Gatekeeper Introduction

The OPA Gatekeeper is a project under the OPA umbrella. Although I did not mention Gatekeeper in the previous tutorial, the technique I described there (using OPA with its sidecar kube-mgmt) is also referred to as Gatekeeper v1.0.

Today, we will get familiar with Gatekeeper v3 (hereafter: just "Gatekeeper", omitting the "v3" part), which builds a Kubernetes admission controller around the policy engine to integrate OPA and the Kubernetes API service.

CRD-Based Policies

Gatekeeper's most significant value is the ability to configure OPA policies dynamically using Gatekeeper's Custom Resource Definitions (CRDs). Custom resources are extensions of the Kubernetes API that allows for the customization of a Kubernetes installation.

CRD-based policies allow for a deeper integration of OPA within the Kubernetes ecosystem: it enables the creation of policy templates for Rego policies, the creation of policies as CRDs, and the storage of audit results on policy CRDs.

How is Gatekeeper different from OPA?

Compared to using OPA with its sidecar kube-mgmt (aka Gatekeeper v1.0), Gatekeeper introduces the following functionality:

An extensible, parameterized policy library
Native Kubernetes CRDs for instantiating the policy library (aka "constraints")
Native Kubernetes CRDs for extending the policy library (aka "constraint templates")
Native Kubernetes CRDs for mutation support
Audit functionality
External data support

This may sound a bit too abstract, so let's get down to the nitty-gritty of how Gatekeeper works.

Install Gatekeeper

First, let's start minikube:

minikube start

For this tutorial, we will deploy a released version of Gatekeeper in our minikube cluster with a prebuilt image:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Note: it's also possible to deploy it with Helm:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts 
helm install gatekeeper/gatekeeper --name-template=gatekeeper --namespace gatekeeper-system --create-namespace

After the deployment, a new namespace gatekeeper-system will be created, and the following resources will be created:

tiexin@mbp ~ $ kubectl get deployments
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
gatekeeper-audit                1/1     1            1           2m20s
gatekeeper-controller-manager   3/3     3            3           2m20s

tiexin@mbp ~ $ kubectl get services
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
gatekeeper-webhook-service   ClusterIP   10.103.86.204   <none>        443/TCP   2m22s

tiexin@mbp ~ $ kubectl get crd
NAME                                                 CREATED AT
assign.mutations. Gatekeeper.sh                       2023-01-01T08:59:54Z
assignmetadata.mutations.gatekeeper.sh               2023-01-01T08:59:54Z
configs.config.gatekeeper.sh                         2023-01-01T08:59:55Z
constraintpodstatuses.status.gatekeeper.sh           2023-01-01T08:59:55Z
constrainttemplatepodstatuses.status.gatekeeper.sh   2023-01-01T08:59:55Z
constrainttemplates.templates.gatekeeper.sh          2023-01-01T08:59:55Z
expansiontemplate.expansion. Gatekeeper.sh            2023-01-01T08:59:55Z
modifyset.mutations. Gatekeeper.sh                    2023-01-01T08:59:55Z
mutatorpodstatuses.status. Gatekeeper.sh              2023-01-01T08:59:55Z
providers.externaldata. Gatekeeper.sh                 2023-01-01T08:59:55Z

Gatekeeper Concepts: Constraints and Constraint Templates

Before moving on to the actual tutorial, let's have a look at two essential concepts of Gatekeeper with concrete examples: constraints and constraint templates.

In short, constraints use constraint templates to inform Gatekeeper what policies to be enforced and how.

I know this sounds a bit confusing, so let's have a look at an example:

The ConstraintTemplate example below contains Rego code which checks if a resource object has a label named "team":

File constraint_template.yaml:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: teamlabel
spec:
  crd:
    spec:
      names:
        kind: TeamLabel
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package teamlabel

        labels := input.review.object.metadata.labels

        has_team {
          labels.team
        }

        violation[{"msg": msg}] {
          not has_team
          msg := "You should have the team label"
        }

The ConstraintTemplate object above doesn't trigger policy enforcement on its own. However, it creates a new custom resource in our cluster of the type TeamLabel. If we want to enforce our TeamLabel policy, we create a constraint by using that new resource type:

File constraint.yaml:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: TeamLabel
metadata:
  name: teampods
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    excludedNamespaces:
    - kube-system
  parameters: {}

This constraint uses the TeamLabel constraint template above to let Gatekeeper enforce our TeamLabel policy for all pods not in the kube-system namespace.

The Gatekeeper service also continually and constantly monitors and audits existing cluster objects to detect policy violations.

How to Use Gatekeeper in Kubernetes

Create files constraint_template.yaml and constraint.yaml with the content in the previous section, and apply them:

kubectl apply -f constraint_template.yaml
kubectl apply -f constraint.yaml

Now let's deploy a simple pod into the default namespace as a test:

kubectl apply -n default -f https://k8s.io/examples/pods/simple-pod.yaml

We will get the following error:

Error from server (Forbidden): error when creating "https://k8s.io/examples/pods/simple-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [teampods] You should have the team label

Let's get in compliance and add a "team" label to the simple pod:

File simple-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    team: test
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80

Create this file with the above content and apply it:

kubectl apply -n default -f simple-pod.yaml

We should encounter no error, and the pod will be created successfully:

tiexin@mbp ~ $ kubectl get pods -n default
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          34s

How to Scale OPA?

So far, we have demonstrated a simple use case of OPA/Kubernetes integration with Gatekeeper.

However, the policies can be much more complicated in real-world scenarios, and you will have multiple policies. How does OPA work at a much larger scale?

Let's talk about three crucial topics on this subject matter:

repository structure
testing policies
other tools

Repository Structure

A well-structured directory helps to make your code more manageable:

.
└── team-label-policy
    ├── README.md
    ├── constraint.yaml
    ├── constraint_template.yaml
    ├── simple-pod.yaml
    ├── src.rego
    └── src_test.rego

In the example above, we put everything that belongs to the team label policy under the folder "team-label-policy", which contains the core files:

src.rego: the OPA Rego code
src_test.rego: the corresponding test cases for our policy Rego
constraint_template.yaml: ConstraintTemplate for our policy. Note that this file also contains the code from src.rego inline, but the OPA tool cannot parse the manifest YAML, so we need to the Rego code to a separate file for testing. If you use this layout for your policies, you must remember to synchronize code changes between the two files.
constraint.yaml: the manifest for a test of the constraint template
simple-pod.yaml: a minimalist pod definition to demonstrate the constraint in practice
README.md

How to Test a Rego Policy

For essential use cases and complicated policies, we should also write tests for those policies.

When writing test coverage for your Gatekeeper policy, you want to consider the following points carefully:

What Kubernetes API resource fields do my policy query? Are any of them optional? Can they appear more than once in a spec?
How many positive test cases do I need to write to ensure my policy will do what I expect?
How many negative test cases do I need to write to ensure my policy will not produce results I do not want?

Policy tests are also written in Rego.

By convention, they live in the same directory as the source file. In our case, we can the policy from the constraint_template.yaml file into src.rego and write tests in the file src_test.rego.

Note the matching package name at the top of each file:

File src.rego:

package teamlabel

# copied from file constraint_template.yaml

labels := input.review.object.metadata.labels

has_team {
  labels.team
}

violation[{"msg": msg}] {
  not has_team
  msg := "You should have the team label"
}

File src_test.rego:

package teamlabel
import future.keywords

test_pod_allowed if {
  results := violation with input as {"review": {"object": {"metadata": {"labels": { "team": "test" }}}}}
  count(results) == 0
}

test_pod_denied if {
  results := violation with input as {"review": {"object": {"metadata": {"labels": {}}}}}
  count(results) > 0
}

Test method names should always begin with the prefix test_.

We can use the OPA command-line tool to evaluate our tests:

tiexin@mbp ~ $ opa test --explain fails src.rego src_test.rego
src_test.rego:
data.teamlabel.test_pod_allowed: PASS (7.852791ms)
data.teamlabel.test_pod_denied: PASS (301.75µs)
--------------------------------------------------------------------------------
PASS: 2/2

If you haven't installed the OPA CLI tool, follow the instructions here.

Helpful Tools

There are some exciting tools (for example, this one) available that can help integrate OPA with your systems and provide ways of writing policies and deploying them across your infrastructure, as well as tools for unit testing, monitoring policy usage, and more. But, even without them, OPA can be managed at a larger scale.

Going from Here

In this 2-part OPA tutorial mini-series, we did two demos on integrating OPA with Kubernetes. If you are interested in OPA and OPA/Kubernetes integration, here is some more reading material for you:

If you like the OPA introduction and the tutorials, please like, comment, and subscribe. See you in the next one!

DevOps with Python: Python ”concurrent.futures“ Concurrency Tutorial - A Real-World Example

Tiexin Guo — Thu, 02 Feb 2023 03:52:39 +0000

Author's note: this blog post ISN'T a beginner's guide to Python or DevOps.

Basic knowledge of Python, DevOps, Kubernetes, and Helm is assumed.

0 Background

0.1 Why Python

Programming languages rise and fall over time.

TIOBE, a Dutch software quality assurance company, has been tracking the popularity of programming languages. According to its programming community index (and its CEO, Paul Jansen, for that matter), Python ranks No.1 now: "for the first time in more than 20 years we have a new leader of the pack: the Python programming language. The long-standing hegemony of Java and C is over."

0.2 Why DevOps with Python

To quote Real Python:

Python is one of the primary technologies used by teams practicing DevOps. Its flexibility and accessibility make Python a great fit for this job, enabling the whole team to build web applications, data visualizations, and to improve their workflow with custom utilities.

On top of that, Ansible and other popular DevOps tools are written in Python or can be controlled via Python.

Plus, I'm a big fan of Python's easy-to-read, no-bracket code style.

One might wonder why easy-to-read is so essential. The 'puter has no problem executing code with ambiguous variable names, lengthy functions, a single file of a thousand (if not thousands) of lines of code, or all of them together, anyway. It will run properly, right?

Well, yes. But to quote Knuth:

Programs are meant to be read by humans and only incidentally for computers to execute.

All the methodologies and ideas, like refactoring, clean code, naming conventions, code smell, etc., are invented so that we, humans, can read the code better, not computers can run it better.

0.3 Why Concurrency

OK, this one is easy:

Because we can.

Jokes aside, the reason is, of course, performance:

Concurrent is faster (usually).

For instance, if you have multiple Helm charts installed in one namespace of a Kubernetes cluster and you want to purge all the Helm releases, of course, you can uninstall them one by one, waiting for the first release to be uninstalled, then start uninstalling the second, etc.

For some applications, the Helm uninstall part can be slow.

Even if for a few simple charts, uninstalling them concurrently can still drastically save time.

Based on a local test, uninstalling three helm charts (Nginx, Redis, and MySQL) one by one takes c.a. 0.8 second, while it takes 0.48s if done concurrently, a whopping 40% reduction.

If the scale of the problem goes up, like you have tens of charts to uninstall and you need to do them in multiple namespaces, the amount of time saved must be addressed.

Next, let's deal with this particular example using Python.

1 The Task

You have multiple teams and developers who share the same Kubernetes cluster as the dev ENV.

To achieve resource segregation, one namespace is assigned to each developer. Each developer needs to do some Helm install to get their apps and dependencies up and running so that they can develop and test them.

Now, since there are many namespaces, many Helm releases in each namespace, and many pods, which take up many nodes, you might wanna optimize the cost by deleting all those pods at the end of the working hours so that the cluster can scale down to save some dollars of the VM cost.

You want some form of automation that uninstalls all releases in some namespaces.

Let's tackle this issue in Python. For demonstration purposes, we will install nginx, redis and mysql into the default namespace, then write some automagic stuff to delete 'em.

Let's go.

2 Preparing the Environment for Testing Our Automation

Not that we are doing test-driven development, but before writing any code, let's create a local environment as a mock of this issue at hand so that we have something to test our automation script.

Here, we use Docker, minikube, and Helm. If you haven't installed them yet, check out the official websites:

Start your local Kubernetes cluster by running:

minikube start

Then, install some Helm charts in the default namespace, which we will use automation to delete later:

# make sure we select the default namespace
kubectl config set-context --current --namespace=default

# add some helm repos and update
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

# install three applications that we will use automation to delete
helm install nginx nginx-stable/nginx-ingress
helm install redis bitnami/redis
helm install mysql bitnami/mysql

Local testing mock done.

3 Non-Concurrent Version

First, let's write some single-thread, non-concurrent code to solve this issue.

We will use the subprocess module to run helm list to get all the releases, run a simple loop over all the releases, and then helm uninstall them. Nothing fancy here; only use Python to run some CLI commands.

Talk is cheap; show me the code:

https://gist.github.com/IronCore864/ca1e74a65f4a97937d93c63c094e9d32

4 Introducing `concurrent.futures`

4.1 `multiprocessing` and `threading`

Before jumping right into concurrent.futures (as advertised in the title of this blog), let's talk multiprocessing and threading for a bit:

The threading module lets you work with multiple threads (also called lightweight processes or tasks) — multiple threads of control sharing their global data space.
multiprocessing is a package that supports spawning processes. The multiprocessing solves the Global Interpreter Lock issue using subprocesses instead of threads.

When choosing between the two, simply put (might not be 100% precise, but that's the gist):

If your task is CPU-intensive, go for multiprocessing (which bypasses the GIL issue by utilizing multiple processes instead of threads).
If your task is I/O-intensive, the threading module should work.

4.2 What is `concurrent.futures`, anyway?

Now that we've got these two modules out of the way, what's concurrent.futures?

It is a higher-level interface to start async tasks and an abstraction layer on top of threading and multiprocessing. It's the preferred tool when you just want to run a piece of code concurrently and don't need the extra functionalities provided by the threading or multiprocessing module's API.

4.3 Learn with an Example

OK, enough theory, let's get our hands dirty and learn by an example, an example from the official documentation:

import concurrent.futures
import urllib.request

URLS = ['http://www.cnn.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        data = future.result()
        print('%r page is %d bytes' % (url, len(data)))

Some observations:

An executor, whatever it is, is required. (Which can either be ThreadPoolExecutor or ProcessPoolExecutor, according to the doc.)
The Executor.submit() method "submits" (or, in plain English, "schedules") the function calls (with parameters) and returns a Future object.
The concurrent.futures.as_completed method returns an iterator over the Future instance.

5 The Concurrent Code to Solve the Task

Once we understand the syntax and get a basic understanding of how it actually works, by copying and pasting from the example and being a little creative, it's easy to convert our non-concurrent version from the previous section into something concurrent. To put it all together:

See the code below:

https://gist.github.com/IronCore864/ad21130aa796d407624805c5342201db

Voila!

Note that the concurrent.futures part is of precisely the same structure as the official concurrent.futures example.

Summary

concurrent.futures cheat sheet:

    # or, with concurrent.futures.ProcessPoolExecutor()
    with futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_objects = {
            executor.submit(some_func, param1, param2 ...): param1 for param1 in xxx
        }

        for f in futures.as_completed(future_objects):
            res = future_objects[f]
            do_something()

Rule of thumb: use ThreadPoolExecutor for I/O-intensive workload and ProcessPoolExecutor for CPU-intensive workload.

If you enjoyed this article, please like, comment, subscribe. See you in the next piece.

A Comprehensive Tutorial on Service Mesh, Istio, Envoy, Access Log, and Log Filtering

Tiexin Guo — Thu, 19 Jan 2023 08:24:24 +0000

First things first, I confess: I haven’t used photos that I took as featured images for a while, and I truly miss that.

In this article, we will briefly introduce Envoy, enable Envoy access log in Istio, play with Envoy's access log filters, and figure out ways to configure Envoy access log filters with Istio.

Some basic knowledge of Istio is expected, but even if you have none, you can follow this tutorial to have a successful local setup.

Without further adieu, let's get started.

1 A Very Short Introduction to Envoy

Envoy is an L7 high-performance proxy and communication bus developed in C++ and designed for large modern service-oriented architectures.

It is a self-contained process designed to run alongside every application server.
At its very core, Envoy is, in fact, an L3/L4 network proxy. A pluggable filter chain mechanism allows filters to be written to perform different TCP/UDP proxy tasks and inserted into the main server.
Envoy supports an additional HTTP L7 filter layer; HTTP filters can be plugged into the HTTP connection management subsystem that performs different tasks such as buffering, rate limiting, routing/forwarding, etc.
When operating in HTTP mode, Envoy supports a routing subsystem capable of routing and redirecting requests based on path, authority, content type, runtime values, etc. This functionality is most useful when using Envoy as a front/edge proxy but is also leveraged when building a service-to-service mesh.

Envoy has many other high-level features, but the ones mentioned above make it perfect to be used as a sidecar proxy in a service mesh.

Two core concepts about Envoy are related to this tutorial:

1.1 HTTP Connection Management

HTTP is such a critical component of modern service-oriented architectures that Envoy implements a large amount of HTTP-specific functionality.

Envoy has a built-in network-level filter called the HTTP connection manager, which translates raw bytes into HTTP level messages and events (e.g., headers received, body data received, trailers received, etc.). It also handles functionality common to all HTTP connections and requests, such as access logging, request ID generation and tracing, request/response header manipulation, route table management, and statistics.

1.2 Access Logging

When used in a service-mesh scenario, for example, in Istio, the simplest kind of logging is Envoy's access logging.

Envoy proxies print access information to their standard output. The kubectl logs command can print Envoy's containers' standard output.

2 A Very Introduction to Istio (And Service Mesh)

2.1 Service Mesh

A service mesh is a dedicated infrastructure layer added to distributed microservices. It allows us to add capabilities transparently like:

observability
traffic management
security

And these are achieved without changing the application code.

As the deployment of distributed services grows in size and complexity, it becomes harder to understand and manage all the services, the requirements of which include discovery, load balancing, failure recovery, metrics, monitoring, etc. And inter-service communication is what makes a distributed application possible. Routing this communication within and across application clusters becomes increasingly complex as the number of services grows.

A service mesh helps reduce the aforementioned while easing the strain on development teams.

It can often handle more complex operational things like A/B testing, canary deployments, rate limiting, access control, encryption, and end-to-end authentication.

2.2 Istio

Istio is an open-source service mesh that layers transparently onto existing distributed applications. Istio's robust features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring – with few or no service code changes. Its powerful control plane brings vital features, including:

Secure service-to-service communication in a cluster with TLS encryption, strong identity-based authentication, and authorization
Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic
Fine-grained control of traffic behavior with rich routing rules, retries, failovers, and fault injection
A pluggable policy layer and configuration API supporting access controls, rate limits, and quotas
Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress

Istio is designed for extensibility and can handle various deployment needs. Istio's control plane runs on Kubernetes. You can add applications deployed in that cluster to your mesh, extend the mesh to other clusters, or even connect VMs or other endpoints outside Kubernetes.

A large ecosystem of contributors, partners, integrations and distributors extend and leverage Istio for a wide variety of scenarios. You can install Istio yourself, or some vendors have products that integrate Istio and manage it for you.

2.3 Istio and Envoy

Istio uses Envoy as proxies, deployed as sidecars. These proxies mediate and control all network communication between microservices. They also collect and report telemetry on all mesh traffic.

It's worth mentioning that, for the sake of unbias, there are other choices of service mesh besides Istio, some of which are even better from certain standpoints. But since Envoy and Istio are closely related, today we will use Istio to demo Envoy's access log settings.

OK, now that we cleared the nomenclature of service mesh and Istio out of the way, let's play with Envoy's access log. We will do this with Istio.

3 Prepare a Local Kubernetes Cluster with Istio

3.1 Prepare a Local Kubernetes Cluster

One of the easiest ways to start a local Kubernetes cluster is to use minikube. Follow the instructions in the official installation documentation, then run the following:

minikube start

3.2 Install Istio

Download Istio, install, then enable istio-injection in the default namespace by running the following commands:

curl -L https://istio.io/downloadIstio | sh -
# your version might differ
cd istio-1.16.1  
# for easier usage
export PATH=$PWD/bin:$PATH 
# here, we use the minimal profile so that the access log isn't enabled by default, which we want to do ourselves
istioctl install --set profile=minimal -y
# enable injection in the default namespace
kubectl label namespace default istio-injection=enabled

3.3 Deploy the Testing Apps

First, let's deploy some sample apps for testing:

kubectl apply -f samples/sleep/sleep.yaml
export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl apply -f samples/httpbin/httpbin.yaml

This set of commands will deploy two applications: one is the "curl" command in a pod, which we will use to send out HTTP requests, and the other is an HTTP server that receives requests, which we will use to demonstrate the access logs of Envoy.

4 Envoy Access Logs in Istio

4.1 Enable Access Logs

Then, let's enable access logs.

Istio offers a few ways to enable access logs. Use of the Telemetry API is recommended:

First, we create a file telemetry.yaml with the following content:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: Envoy

Then apply it:

kubectl apply -f telemetry.yaml

4.2 Test

First, let's send a request from sleep to httpbin:

kubectl exec "$SOURCE_POD" -c sleep -- curl -sS -v httpbin:8000/status/418

If we check the httpbin's log, we can see something similar to the following:

kubectl logs -l app=httpbin -c istio-proxy
[2020-11-25T21:26:18.409Z] "GET /status/418 HTTP/1.1" 418 - via_upstream - "-" 0 135 3 1 "-" "curl/7.73.0-DEV" "84961386-6d84-929d-98bd-c5aee93b5c88" "httpbin:8000" "127.0.0.1:80" inbound|8000|| 127.0.0.1:41854 10.44.1.27:80 10.44.1.23:37652 outbound_.8000_._.httpbin.foo.svc.cluster.local default

5 Envoy Access Log Filter

Now that we have enabled access logs for Envoy, let's play with it.

5.1 The Task

Imagine the following situation: your application has some endpoints, for example, /status, /liveness, and /readiness, which you don't want logs because there might be multiple requests per minute. These status check logs could not be a good use of logging resources.

Or, you may want to customize your access logs to allow different requests and responses to be written to separate log files.

Can we achieve that?

Yes.

Envoy supports several built-in access logs and extension filters registered at runtime. Now let's try to demo the log filter functions:

5.2 Download Envoy And Prepare a Config File

The easiest way to play with Envoy's configuration is by running it locally instead of as a sidecar as part of Istio. So let's do this:

Install Envoy:

brew update
brew install envoy

Download the demo config from here

Then let's start it:

envoy -c envoy-demo.yaml

Then, open another tab, and verify that Envoy is running on port 10000:

curl -v localhost:10000

If we go back to the tab where Envoy is running, we can see the access log from Envoy, which will look similar to the following:

[2023-01-19T03:10:59.085Z] "GET / HTTP/1.1" 200 - 0 17304 747 467 "-" "curl/7.85.0" "4b35db72-baf8-4730-885e-3e628757f0e3" "www.envoyproxy.io" "34.143.223.220:443"

5.3 Envoy Log Filter

Envoy provides a bunch of log filters, for example, status code filter (as the name suggests, filter by status code), header filter, etc. Read the official doc here for more details.

The documentation could be more detailed; it doesn't provide examples of the usage of each filter. But that won't block a senior DevOps engineer. Based on some simple searches on Google (and GitHub), let's try this piece of config:

...
          access_log:
          - name: Envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
            filter:
              header_filter:
                header:
                  name: :Path
                  string_match:
                    exact: /status
...

It's not hard to understand: it works on the header, and if the path matches something, the filter catches it.

If we add this piece of code into the file envoy-demo.yaml, and if we access the /status URL:

curl -v localhost:10000/status

There will be some logs like:

[2023-01-19T03:23:08.054Z] "GET /status HTTP/1.1" 404 - 0 3413 776 690 "-" "curl/7.85.0" "e507c86f-004f-4cba-b622-2004c7cf97c7" "www.envoyproxy.io" "34.126.184.144:443"

But if we access URLs other than /status, such as /:

curl -v localhost:10000

We get no logs.

OK, so the log filter is working now. Except that we want to filter out logs like /status.

5.4 More on Envoy Log Filters

OK, let's go back to the official doc and some google searches.

It's not hard to see that some filters like "and_filter" are helpful. Let's try that:

...
            filter:
              and_filter:
                filters:
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /status
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /liveness
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /readiness
                      invert_match: true
...

We can use and_filter to combine a bunch of filters, and we can negate the matching result by using invert_match.

This config updated in the envoy-demo.yaml will only show logs if the URL doesn't match /status, /liveness, or /readiness.

OK, we have achieved our goal: we can use filters to control Envoy's access logging. In the example above, we used and_filter and header_filter to filter out specific unwanted log entries. If you mix and match all Envoy's filters, you can quickly achieve things like "direct certain logs to a specific file", "only log 4xx requests", etc.

But (and this is a big but), the above config is only for local usage or running Envoy as a standalone process.

How to put those filter settings into a sidecar Envoy?

6 Telemetry API

As previously mentioned, Istio offers a few ways to enable access logs. We used Telemetry API in section 4.1, so let's see if we can customize Telemetry API to achieve the log filtering goal.

Telemetry defines how the telemetry is generated for workloads within a mesh. It has a feature AccessLogging.Filter where you can specify an expression in a string to achieve goals like response.code >= 400 or connection.mtls && request.url_path.contains('v1beta3').

According to the official documentation (link above), the expression should be a CEL expression (read more details on the language definition here.

So, if we convert the filters we wrote above into a CEL expression, it would be like this:

expression: "request.url_path != '/status' && request.url_path != '/liveness' && request.url_path != '/readiness'"

If we update the telemetry.yaml with the following content:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
        - name: Envoy
      filter:
        expression: "request.url_path != '/status' && request.url_path != '/liveness' && request.url_path != '/readiness'"

And apply it:

kubectl apply -f telemetry.yaml

Then the istio-proxy sidecar (Envoy) will only log entries where the request URL path isn't /status, /liveness, or /readiness.

7 EnvoyFilter

EnvoyFilter provides a mechanism to customize the Envoy configuration. Use EnvoyFilter to modify values for certain fields, add specific filters, or even add entirely new listeners, clusters, etc. This feature must be used carefully, as incorrect configurations could destabilize the entire mesh. Unlike other Istio networking objects, EnvoyFilters are additively applied. Any number of EnvoyFilters can exist for a given workload in a specific namespace. The order of application of these EnvoyFilters is as follows: all EnvoyFilters in the config root namespace, followed by all matching EnvoyFilters in the workload's namespace.

We can use EnvoyFilter to configure Envoy access logs, although officially, "Istio Telemetry API provides a first-class way to configure access logs and traces. It is recommended to use that method."

Nevertheless, let's look at EnvoyFilter and how to use that.

First, let's delete the Telemetry we created before to disable the access log:

kubectl delete -f telemetry.yaml

Then, following the format in this doc, we can create a file envoyfilter.yaml with the following content:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: access-log
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: "Envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          access_log:
          - name: Envoy.file_access_log
            typed_config:
              "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
              path: /dev/stdout
              format: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% \"%RESP(GRPC-STATUS)% %RESP(GRPC-MESSAGE)%\" %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"
            filter:
              and_filter:
                filters:
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /status
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /liveness
                      invert_match: true
                - header_filter:
                    header:
                      name: :Path
                      string_match:
                        exact: /readiness
                      invert_match: true

We create an object of the "EnvoyFilter" kind, and the content looks the same as Envoy's configuration file.

If we apply it:

kubectl apply -f envoyfilter.yaml

Then the istio-proxy sidecar (Envoy) will only log entries where the request URL path isn't /status, /liveness, or /readiness, just like the Telemetry way.

8 Teardown

minikube delete

Summary

In this tutorial:

We introduced Envoy, service mesh, and Istio.
Then, we created a local Kubernetes cluster and installed Istio inside it.
We enabled Envoy access logs in Istio via Telemetry and played with Envoy's configurations to achieve log filtering.
We also looked at two ways of setting log filtering for the Envoy sidecar in Istio (Telemetry and EnvoyFilter).

I know not all people have the need to fine-tune Envoy’s access logging, but if you do, you will find it hard to get some examples off the internet. That’s one of the reasons I decided to publish my experience as a blog: if you are searching hard to figure out how to write filters, and_filter, EnvoyFilter, or Telemetry API AccessLogging, CEL expression syntax, you are not alone, and I hope this article helps.

For people who are not so crazy about Envoy access logging, this tutorial still serves as a very good introduction to service mesh/Istio/Envoy.

If you like this article, please like, comment, subscribe. See you in the next one!

App-Centric Configuration: Adding More Value to Your Life

Tiexin Guo — Wed, 30 Nov 2022 01:55:06 +0000

DevStream's Story

About nine months ago, DevStream was first publicly released. Since then, it has evolved a lot. If this is the first time you have come across DevStream, maybe read this blog for a quick overview.

In the current incarnation (v0.9), every single DevOps tool (including integrations of tools and CI/CD pipeline setup for apps) is treated as a "Tool", which is a DevStream concept. So, if you would like to define a DevOps platform that does the following:

repository scaffolding
continuous integration w/ GitHub Actions
install Argo CD as the tool for continuous deployment
continuous deployment w/ Argo CD

Your DevStream tools.yaml config file would look similar to this:

tools:
- name: repo-scaffolding
  instanceID: myapp
  options:
    destinationRepo:
      owner: [[ githubUser ]]
      repo: [[ app ]]
      branch: main
      repoType: github
    sourceRepo:
      org: [[ githubUser ]]
      repo: dtm-scaffolding-python
      repoType: github
    vars:
      ImageRepo: [[ dockerUser ]]/[[ app ]]
      AppName: [[ app ]]
- name: githubactions-python
  instanceID: default
  dependsOn: [ repo-scaffolding.myapp ]
  options:
    owner: [[ githubUser ]]
    repo:  [[ app ]]
    language:
      name: python
    branch: main
    docker:
      registry:
        type: dockerhub
        username: [[ dockerUser ]]
        repository: [[ app ]]
- name: helm-installer
  instanceID: argocd
- name: argocdapp
  instanceID: default
  dependsOn: [ "argocd.default", "githubactions-python.default" ]
  options:
    app:
      name: [[ app ]]
      namespace: argocd
    destination:
      server: https://kubernetes.default.svc
      namespace: default
    source:
      valuefile: values.yaml
      path: helm/[[ app ]]
      repoURL: ${{repo-scaffolding.myapp.outputs.repoURL}}

A quick explanation - what DevStream does with this config is the following:

install Argo CD;
repository scaffolding for myapp;
CI/CD for myapp.

It looks nice and works like a charm.

However, suppose you have more than one app to worry about (more often than not, that would be the case in the microservice era). In that case, things start to get a bit complicated: you'd have to repeat the githubactions-golang and argocdapp sections as many times as the number of apps you manage.

That is to say, to manage ten apps/microservices, your DevStream YAML file may grow to well past 300 lines of config.

Today, we are happy to announce that it's no longer the case. We've simplified it. By a big margin. With the release of v0.10.

Read on.

Adding Value to Your Life

A big config is definitely harder to read and maintain, and we don't like that. That's why we decided to solve this problem in the first place.

However, we waited to get right into it.

Instead, we started to think about our market positioning and value proposition. What is DevStream anyway? What problems does it solve? Why would other engineers want to use it instead of building stuff manually or purchasing one-stop DevOps platforms?

If we could precisely answer these questions, we would know how to improve it to the next level.

The discussion went on and on. The whole team, including our CEO, discussed multiple days. We've put tens of hours of thought and debate into it, until we figured out where exactly DevStream can add value to your life:

installing DevOps tools (but that's not the point at all)
integrating DevOps tools and building engineering platforms (now we're talking)
application lifecycle/software development lifecycle management:
- repository bootstrapping
- continuous integration pipelines, integrating best practices, typical stages, and steps into it
- continuous deployment pipelines, integrating helm chart/Dockerfile best practices
- ...

Lightbulb: Apps? Apps!

Since many values DevStream could bring are around applications/microservices and their lifecycle management, why not simply build configurations based on that?

We call this new thought "app-centric" configuration, and let's get right into it:

Backward Compatibility

First things first, the previous "tools" config still works. We wouldn't want to break that. That is to say, if you copy-paste the first code snippet from the beginning of this article, it's supposed to work without any problem, just like previous versions.

(Much) Shorter, Simpler, Easier, While Achieving the Same

You'd be surprised that the config below does exactly the same thing as the code snippet at the very beginning of this article.

apps:
- name: myapp
  spec:
    language: python
    framework: django
  repo:
    url: github.com/devstream-io/myapp
  repoTemplate:
    url: github.com/devstream-io/dtm-scaffolding-python
  ci:
    - type: githubactions
  cd:
    - type: argocdapp

Is this magic? How'd we do that?

First, we created a new abstraction that is called "apps". Each app corresponds to a real-world application or microservice that you manage.

Secondly, we simplified the configurations as much as possible with five small sections:

spec: language, framework related to the application, which is shared information with other sections
repo: where to create/bootstrap the repository for the app
repoTemplate: the template used to bootstrap the app's repo
ci: setting up continuous integration for this app
cd: setting up continuous deployment for this app

Last but not least, more "defaults" and "best practices" are integrated, by default, into DevStream, so that you don't have to override most of the configs.

Neat, right? I know.

One File to Rule Them All

Putting It All Together:

config:
  state:
  backend: local
  options:
    stateFile: devstream.state

tools:
- name: helm-installer
  instanceID: argocd

apps:
- name: myapp1
  spec:
    language: python
    framework: django
  repo:
    url: github.com/devstream-io/myapp1
  repoTemplate:
    url: github.com/devstream-io/dtm-scaffolding-python
  ci:
  - type: githubactions
  cd:
  - type: argocdapp
- name: myapp2
  spec:
    language: golang
    framework: gin
  repo:
    url: github.com/devstream-io/myapp2
  repoTemplate:
    url: github.com/devstream-io/dtm-scaffolding-golang
  ci:
  - type: githubactions
  cd:
  - type: argocdapp

I hope you enjoy this new feature. Have fun experimenting tools together with apps!

DEV Community: Tiexin Guo

Container Security Scanning: Vulnerabilities, Risks and Tooling

Container Vulnerabilities

Container Security Scanning

Hard-Coded Secrets and Misconfigurations in Docker Images

ggshield

SecretScanner

trivy

Docker Image Scanning for CVEs

SCA: Software Composition Analysis

Container Runtime Security

Summary

How to Handle Secrets in Jupyter Notebooks

Entering Passwords on the Fly in Jupyter Notebooks: getpass

Reading Secrets from a File Uploaded to JupyterHub

Encrypt Jupyter Notebook Secrets Files in Git

The Ultimate Solution: Access Secret Managers from Jupyter Notebooks

Using Secret Managers in Jupyter Notebooks

Using Cloud-Based Jupyter Service to Access Secret Managers

Handling Secrets in Kaggle Notebooks

Summary

Open-Source Software Security

An Introduction to Open Source Software Security

What is Open Source Software Security

Why You Should Care about Open Source Software Security

The Complexity of Open Source Dependencies

Vulnerability Scanning

SBOM, and Software Composition Analysis

Licensing

Conclusion

FAQ

Which is better for security, open source or proprietary software?

How do you make open-source software secure?

What is the difference between Software Composition Analysis (SCA) and Open Source Software (OSS)?

The New Frontier in Cybersecurity: Embracing Security as Code

How We Used to Handle Security

Understanding DevSecOps and Security as Code

DevSecOps: Shift Security to the Left of the SDLC

What is Security as Code (SaC)?

The Importance and Benefits of Security as Code (SaC)

Key Components of Security as Code

Best Practices for Security as Code

Security-First Mindset for Security as Code/DevSecOps

Code Reviews + Automated Scanning

Continuous Monitoring, Feedback Loops, Knowledge Sharing

Security as Code: What You Should Keep in Mind

Conclusions and Next Steps

F.A.Q.

What is Security as Code (SaC)?

What is the main purpose of Security as Code?

What is the relationship between Security as Code and DevSecOps?

What is the difference between DevSecOps and secure coding?

What is Infrastructure as Code (IaC) security?

How to Handle Secrets in Helm

Helm Charts and Helm Secrets

Tutorial: The helm-secrets Plugin

A Quick Introduction to the helm-secrets Plugin

How helm-secrets works

Example 1: Helm-secrets with SOPS

Example 2: helm-secrets with Cloud Secrets Managers

Simplifying Continuous Deployment Integration with helm-secrets

External Secrets Operator

A Quick Introduction to External Secrets Operator

How External Secrets Operator Works

External-Secrets Helm Chart Example

Seamless Integration of External Secrets Operator with Continuous Deployment Tools

Vault Secrets Operator for Kubernetes Secrets and Helm Secrets

How to Automatically Restart Pods When Secrets Are Updated

Summary

F.A.Q.

Handling Secrets with AWS Secrets Manager

1 What is AWS Secrets Manager?

2 What Secrets Can Be Stored in Secrets Manager?

3 Secrets Manager Features

4 How Secrets Manager Works under the Hood

5 Creating a Secret in Secrets Manager

5.1 Create/Update

5.2 Secret Versions

5.3 List/Find

5.4 Delete

Entering Passwords on the Fly in Jupyter Notebooks: `getpass`

5.1 The `skeleton` Folder

5.2 The `template.yaml` File

4 Introducing `concurrent.futures`

4.1 `multiprocessing` and `threading`

4.2 What is `concurrent.futures`, anyway?