DEV Community: Miguel

ChatGPT creating OSS security rules and plugins

Miguel — Mon, 27 Feb 2023 09:44:31 +0000

Can OpenAI ChatGPT become a contributor for an open source project like Falco? Is this the end of GitHub Copilot?

ChatGPT is SO trendy that I overheard my grandma talking about it to her friends the other day. We’re seeing more and more uses of this AI for real world applications. That made us think…

Falco, the first runtime security project to join CNCF as an incubation-level project, needs contributors. Would OpenAI’s ChatGPT language model be able to write security rules or plugins for Falco, and become a contributor?

ChatGPT as a potential new Falco partner

Jokes aside, in this article we explore the viability of ChatGPT as a tool for organizations to create effective and efficient security rules. Would those new rules in Falco ensure their containers and hosts are protected from potential threats?

What is runtime security for ChatGPT?

In order to begin, we need to understand what runtime security is. Rather than using a search engine, we will ask ChatGPT in order to speed up the process.

According to ChatGPT:

“Runtime security involves monitoring and protecting software systems in real-time as they are running, detecting, and responding to security threats, and implementing mitigation techniques to prevent further damage. The goal of runtime security is to identify and respond to security threats and vulnerabilities before they can be exploited and cause harm.”

A good definition, but let’s see if we can go deeper.

What is Falco for ChatGPT?

ChatGPT summarized the open source project Falco in a clear and concise manner.

Rather than copying the information from the landing page of falco.org, ChatGPT provided useful context as to how Falco utilizes eBPF to achieve low-overhead when detecting security threats from data collected within the Linux Kernel.

At this point, we understand what runtime security is, and how Falco can be used to detect anomalous runtime security issues. Now that we are familiar with open source Falco, let’s ask ChatGPT to write us some useful Falco rules.

Asking ChatGPT to create a Falco rule

Now, let’s ask ChatGPT if the language model is capable of writing OSS Falco security rules.

Based on the below screenshot, does ChatGPT looks like a useful contributor to the Falco community?

At this point, we are happy with the answer that was returned.

There was a correctly-formatted Falco rule and the language model also returned some added context as to how the rule will work.

My only concern is that the first rule they created is similar to a rule that already exists in the Falco community rules feed:

- rule: Terminal shell in container
  desc: A shell was used as the entrypoint/exec point into a container with an attached terminal.
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
    and container_entrypoint
    and not user_expected_terminal_shell_in_container_conditions
  output: >
    A shell was spawned in a container with an attached terminal (user=%user.name user_loginuid=%user.loginuid %container.info
    shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline pid=%proc.pid terminal=%proc.tty container_id=%container.id image=%container.image.repository)
  priority: NOTICE
  tags: [container, shell, mitre_execution, T1059]Code language: Perl (perl)

The above Falco community rule includes different use cases; proc.name is not just sh, it is a long list which is contained in shell_procs. As a result, this would lead to fewer false/positive detections and reduce the attackers’ chances of bypassing the rule. If the rules are too generic, it can potentially capture expected behavior.

When asking our questions to ChatGPT, we need to be more precise to generate an accurate rule. For example, we would ask ChatGPT to create a Falco rule that detects suspicious login activity on a Linux workstation between certain hours of the day.

Again, we like how the rule looks.

Since Falco is designed to handle Linux system calls, there is no need to explicitly mention the workstation OS type. However, ChatGPT nicely mentioned that the rule triggers for activity on Linux workstations because we specifically requested this. We will copy the code snippet and paste it below so that we can dissect it further:

- rule: Detect suspicious login activity during off-hours
  desc: Detects login sessions initiated during off-hours on a Linux workstation
  condition: (evt.time > "2022-12-31T02:00:00.000Z" and evt.time < "2022-12-31T07:00:00.000Z") and (evt.type=execve and evt.argc=3 and evt.argv[2]=login)
  output: Suspicious login activity detected during off-hours: user=%user.name command=%proc.cmdline
  priority: WARNINGCode language: Perl (perl)

The Falco rule uses the below system call activity:

evt.time – This is the event timestamp. It’s between T02:00 (2 a.m.) and T07:00 (7 a.m.).
evt.type – This is the name of the event, for example, ‘open’ or ‘read.’ In this case, it’s execve. The execve event executes the program referred to by pathname.

If you are ever unsure about a certain argument used, what it means, or how to use it going forward, you can ask ChatGPT to elaborate on its findings without re-writing the entire question.

Since ChatGPT is a language model, it does a great job of not just providing rules, but also providing clarity on its findings. With this additional context provided by ChatGPT, we are happy with how this rule turned out.

Since we don’t have any business need for this specific rule, let’s use ChatGPT to solve some real business problems.

ChatGPT, MITRE ATT&CK, and Falco

Continuing the conversation, we got more technical with ChatGPT and tried to combine two areas of expertise: Falco and MITRE.

The MITRE ATT&CK framework for Enterprise environments is BIG! As a result, it can be hard to provide extensive coverage of all Tactics, Techniques, and Sub-Techniques for Linux Systems.

Since ChatGPT can read and interpret large values of operational data, it speeds up the process of building Falco rules to better align with this widely-used risk framework.

In the Falco community rules feed, there was no existing rule aligned to the Technique ID ‘T1529.’ For this technique ID, the adversaries may shutdown or reboot the workstation to interrupt access to workstations, or aid in the destruction of those systems. When requesting a rule that detects system shutdown or reboot, we also want to request the appropriate tagging for rules alignment with the MITRE ATT&CK framework. Surprisingly, ChatGPT answered with an incorrect tactic and technique associated with that technique ID.

The technique Cloud Service Dashboard is assigned to the Tactic ‘Discovery’ and the associated Technique ID T1538. Whereas, the technique ID T1529 is associated with shutdown/reboot activity, this would be aligned with the Tactic ‘Impact.’

For the first time, ChatGPT made an obvious mistake in its answer. When we confronted ChatGPT, it immediately apologized and provided an amended answer that looks more like the Falco rule we would expect.

This regained my trust in ChatGPT becoming an approved Falco contributor.

However, since we cannot guarantee that ChatGPT is going to return the correct rule, we also need to validate that the rule conditions are valid.

Again, I’ve pasted the findings into the following snippet field for further inspection. As mentioned by ChatGPT, this rule checks for execve events where the second argument (evt.argv[1]) contains either shutdown or reboot. This indicates that the process is attempting to shut down or reboot the system, which is a technique used to disrupt normal system operation and, therefore, correctly aligns with the MITRE tactic and technique.

- rule: Detect T1529 - System Shutdown/Reboot
  desc: Detects attempts to shut down or reboot the system
  condition: (evt.type=execve and (evt.argv[1] contains "shutdown" or evt.argv[1] contains "reboot"))
  output: "Detected attempt to shut down or reboot the system. T1529 - System Shutdown/Reboot detected"
  priority: WARNING
  tags: [tactic=impact, technique=T1529, technique_id=T1529]Code language: Perl (perl)

So far, we have learned that we cannot rely on ChatGPT to contribute Falco rules without being vetted by an experienced Falco user.

That said, ChatGPT has quickly contributed rules that can be used to address regulatory frameworks and/or risk frameworks such as MITRE ATT&CK. The injected tags allow users to categorize and track detections of this technique within your security management tooling.

How to detect cryptomining with ChatGPT and Falco

The rules we have created so far are fairly simplistic. In order to test the true power of ChatGPT, we need to ask it for help creating more complex Falco rules involving additional abstractions such as Macros and Lists.

An example that we were working on recently was the creation of a small list of known cryptomining binaries for a CNCF Livestream. We would like to see how ChatGPT addresses this request.

We were disappointed with this response.

While the syntax is valid, the default approach from ChatGPT is always to list the process names within the Falco rule, rather than creating a list of known binaries, and mapping this to the Falco rules via a referenced Macro.

We can ask ChatGPT to specifically reference the binaries in the List.

Funnily, ChatGPT was even more confused by this instruction to the point where it started appending syntax that is foreign to the Falco rules syntax.

At this point, the rule would no longer work and ChatGPT is losing credibility as a valid contributor to the Falco project.

As an experienced Falco user, I had to explain that ChatGPT misunderstood my request and that further evaluation is required. It’s not that ChatGPT is unable to answer the request, but it can misunderstand certain aspects of the request depending on our phrasing.

That’s why your request might require further fine tuning, but we can see that ChatGPT got there in the end.

ChatGPT has given us a correctly-formatted Falco rule, which is a great foundation for further development. However, the rule is certainly not foolproof.

There are many examples of cryptomining binaries other than ‘xmrig’ – though xmrig is certainly the most common example. The value here is creating an extensive, up-to-date list of all common binaries so we can provide as much security coverage as possible. We mention some of these binaries in the following Falco blog.

Can ChatGPT create Falco plugins?

Don’t get disappointed. Let’s see if ChatGPT is able to help us create a plugin for Falco.

It’s super important to understand how ChatGPT responds to generic commands. When we asked if ChatGPT can create Falco plugins, it said, “I do not have the ability to write or compile code. However, I can assist you in writing the code for a Falco plugin.”

It’s also worth noting that ChatGPT explains the supported protocol (gRPC) and the languages that support it (C++, Go, or Python).

In that case, we just need to ask for guidance in how to write a Falco plugin. We need the request to be for a specific service for this request to be effective. Since LastPass is in the news lately, we will ask ChatGPT to help us create a LastPass Plugin.

Amazingly, the below Python code snippet was provided to help configure a LastPass plugin with the appropriate gRPC protocol that we mentioned previously.

import grpc
from falco_proto import event_pb2
from falco_proto import event_pb2_grpc

class LastPassPlugin(event_pb2_grpc.EventServiceServicer):
    def HandleEvent(self, request, context):
        event = request.event
        if event.event_type == "executed" and event.output.find("lastpass-cli") != -1:
            print("LastPass CLI was executed")
        return event_pb2.HandleEventResponse()

server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
event_pb2_grpc.add_EventServiceServicer_to_server(LastPassPlugin(), server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()Code language: Perl (perl)

This code sets up a gRPC server and implements a custom HandleEvent method that is called whenever a Falco event occurs. The method checks if the event is an “executed” event and if the output contains the string “lastpass-cli.” If both conditions are met, it prints a message indicating that LastPass CLI was executed.

Note that this is just a basic example. ChatGPT clearly explains that this was provided as guidance and therefore you will likely need to modify the code to meet your specific requirements.

For more information on creating Falco plugins, we would recommend referring to the official Falco documentation and the gRPC protocol documentation.

Can ChatGPT contribute to the Falco project?

Unfortunately, no!

As ChatGPT explained to us, it can help with the rule creation. But as an AI language model, it is not authorized to create pull requests. As a result, ChatGPT cannot be officially included as a contributor to the open source project. However, project contributors and community members can rely on ChatGPT to validate their rule formatting, identify discrepancies in misconfigured rules, as well as provide insights on how a rule should be formatted to address a framework requirement.

Conclusion

ChatGPT is a powerful language model that can assist in creating Falco security rules. With its vast knowledge of various topics and its ability to generate text, it can provide helpful guidance and examples of how to create a rule that detects a specific threat. However, while it can be a valuable resource, ChatGPT should not be trusted to fully automate the creation of security rules.

The accuracy and relevance of the information it provides can be limited by its training data and its knowledge cutoff, and it may not have the expertise or context to make informed decisions about the specific security needs of an organization. Additionally, security rule creation is an ongoing process that requires constant monitoring, tuning, and updating to keep up with new threats and changes in technology.

Therefore, it is an option to use ChatGPT and consult security experts to verify and refine the rules before deploying them in a production environment.

Getting started with kubectl plugins

Miguel — Wed, 18 Jan 2023 16:46:11 +0000

Let's dig deeper into this list of Kubectl plugins that we strongly feel will be very useful for anyone, especially security engineers.

Kubernetes, by design, is incredibly customizable. Kubernetes supports custom configurations for specific use case scenarios. This eliminates the need to apply patches to underlying features. Plugins are the means to extend Kubernetes features and deliver out-of-the-box offerings.

What are Kubernetes Plugins?

Users can install and write extensions for kubectl, the Kubernetes command line tool.

By observing the core kubectl commands as essential building blocks for interacting with a Kubernetes cluster, a cluster administrator can think of plugins as a means of utilizing these building blocks to create more complex behavior.

Plugins extend kubectl with new sub-commands, allowing for new and custom features not included in the main distribution of kubectl.

Why are plugins useful for security operations?

Kubernetes plugins provide countless security benefits to the platform. Incident responders can develop additional functionality “on the fly” in their language of choice.

Since Kubernetes features often fall short in cases where businesses need to achieve “out-of-scope” functionality, teams will often need to implement their own custom operations.

Potential security considerations for Kubernetes plugins

While custom implementations add functionality that is not necessarily provided out-of-the-box with kubectl, these plugins are not always as secure as we would like them to be. This article aims to address the most common or useful Kubernetes plugins for improving your security posture.

Managing plugins with Krew

Krew is a plugin manager maintained by the Kubernetes Special Interest Group (SIG) CLI community. Krew makes it easy to use kubectl plugins and helps you discover, install, and manage them on your machine. It is similar to tools like apt, dnf, or brew. Today, over 200 kubectl plugins are available on Krew - and that number is only increasing. Some projects are actively used and some get deprecated over time, but are still accessible via Krew.

Command to install kubectl plugins via Krew:

kubectl krew install <PLUGIN_NAME>

Kubectl plugins available via the Krew plugin index are not audited, which can cause a problem in the supply chain. As mentioned earlier, the Krew plugin index houses hundreds of kubectl plugins:

https://krew.sigs.k8s.io/plugins/

When you install and run third-party plugins, you are doing this at your own risk. At the end of the day, kubectl plugins are just arbitrary programs running in your shell.

Finally, we want to share our top 15 kubectl plugins that will improve your security posture in your Kubernetes cluster.

1. Stern plugin

Link to GitHub Repository

Stern is a kubectl plugin that works a lot like ‘tail -f’ in Linux. Unlike kubectl log -f, which has its own limitations around input parameters, Stern allows you to specify both the Pod ID and the Container ID as regular expressions.

Any match will be followed and the output is multiplexed together, prefixed with the Pod and Container ID, and color-coded for human consumption (colors are stripped if piping to a file).

You can install Stern with the below Krew command:

kubectl krew install stern

Command to tail an appname in Stern:

kubectl stern appname

This will match any pod containing the word service and listen to all containers within it. If you only want to see traffic to the server container, you could do:

kubectl stern --container

This will stream the logs of all the server containers, even if running in multiple pods.

One interesting security use case for the Stern plugin is to look at authentication activity to your Kubernetes clusters. To show the authentication activity within the last 15 minutes with relevant highlighted timestamps, run the following command:

kubectl stern -t --since 15m auth

2. RBAC-tool

Link to GitHub Repository

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. The RBAC-tool simplifies querying and the creation of RBAC policies.

You can install the RBAC-tool with the below Krew command:

kubectl krew install rbac-tool

If you are unfamiliar with how RBAC roles are assigned to different Kubernetes components, the visualization command generates an insightful graph of all RBAC decisions.

kubectl rbac-tool viz --cluster-context nigel-douglas-cluster

The above command scans the cluster with the kubeconfig context 'nigel-douglas-cluster.' These graphs are useful for showing a visual before-and-after of permissions assigned to service accounts.

There are multiple commands other than ‘viz’ provided by the RBAC-tool plugin. The most useful is the ‘who-can’ command. This shows which subjects have RBAC permissions to perform an action denoted by ‘VERB’ (Create, Read, Update, or Delete) on an object.

To see who can read a secret resource by the name ‘important-secret,’ run the below command:

kubectl rbac-tool who-can get secret/important-secret

3. Cilium Plugin

Link to GitHub Repository

Cilium is a network security project that continues to grow in popularity due to its powerful eBPF dataplane. Since Kubernetes is not designed with any specific CNI (Network) Plugin in mind, it can be deciduous trying to manage the Cilium agent via kubectl. That’s why the Cilium team released the Cilium kubectl plugin.

You can install the Cilium plugin with the below Krew command:

kubectl krew install cilium

As a basic first step, you can do a connectivity check for a single node powered by Cilium networking via the below command:

kubectl cilium connectivity test --single-node <node>

This doesn’t just provide improved operational visibility - it’s incredibly beneficial to network security engineers. For instance, if Cilium is unable to communicate with core components such as ‘Hubble,’ this will show-up in the connectivity test.

Hubble provides network, service, and security observability for Kubernetes. Being able to quickly diagnose a connection error, such as “connection refused,” improves the overall visibility of threats and provides the centralized network event view required to maintain regulatory compliance. If you want to dig deeper into network policies, discover how to prevent a Denial of Service (DoS) attack on Kubernetes.

4. Kube Policy Advisor

Link to GitHub Repository

The kube-policy-advisor plugin suggests PodSecurityPolicies and Open Policy Agent (OPA) Policies for your Kubernetes cluster. While PodSecurityPolicies are deprecated, and therefore should not be used, OPA is very much a recommended tool for admission controller.

You can install advise-policy with the below Krew command:

kubectl krew install advise-policy

This kubectl plugin provides security and compliance checks for Kubernetes clusters. It can help identify potential security risks and violations of best practices in a cluster's configuration, and provide recommendations for how to remediate those issues. Some examples of the types of checks that kube-policy-advisor can perform include:

Ensures pods are running with minimal privileges and are not granted unnecessary permissions.
Checks that secrets and other sensitive data are not stored in plain text or checked into source control.
Verifies that network policies are in place to protect against unauthorized access to resources.
Evaluates the security of container images and ensures that they come from trusted sources.

In Kubernetes, Admission Controllers enforce semantic validation of objects during create, update, and delete operations. With OPA, you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server.

kube-policy-advisor is a tool that makes it easier to create OPA Policy from either a live K8s environment or from a single .yaml file containing a pod specification (Deployment, DaemonSet, Pod, etc.). In the below command, the plugin inspects any given namespace to print a report or OPA Policy.

kubectl advise-policy inspect --namespace=<ns>

Note: If you do not enter a given namespace, it will generate the OPA Policy for all network namespace.

By using kube-policy-advisor, you can help ensure that your Kubernetes cluster is secure and compliant with best practices, which can help protect your applications and data from potential threats.

5. Kubectl-ssm-secret

Link to GitHub Repository

The kubectl-ssm-secret plugin allows admins to import or export their Kubernetes Secrets to or from an AWS SSM Parameter Store path. A Kubernetes Secret is sensitive information – such as a password or access key – that is used within a Kubernetes environment. It’s important to be able to safely control these sensitive credentials when transmitting between Kubernetes and AWS cloud.

You can install the ssm-secret plugin with the below Krew command:

kubectl krew install ssm-secret

Secrets are not unique to Kubernetes, of course. You use Secrets’ data in virtually every type of modern application environment or platform. In the case of the ssm-secret plugin, all parameters found under a given parameter store path can be imported into a single kubernetes secret as “StringData.”

This is incredibly useful if you are reprovisioning clusters or namespaces and need to provision the same secrets over and over. Also, it could be useful to backup/restore your LetsEncrypt or other certificates.

If an AWS parameter at path /foo/bar contains a secret value, and the parameter /foo/passwd contains a secure password, we can view the keys and values in parameter store using the kubectl ssm-secret list subcommand:

kubectl ssm-secret list --ssm-path /foo

Those output parameters can then be imported with the following import command:

kubectl ssm-secret import foo --ssm-path /foo

Security considerations

You must specify a single parameter store path for this plugin to work. It will not recursively search more than one level under a given path. As a result, the plugin is highly opinionated, and users run the risk of failing to import/export secrets to the correct path if they don’t track these paths correctly.

6. Kubelogin

Link to GitHub Repository

If you’re running Kubectl versions v.1.12 or higher, Kubelogin (also known as kubectl-login) is a useful security plugin for logging into clusters via the CLI. It achieves this through OpenID Connect providers like DEX. OpenID Connect is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.

You can install the kubectl-login plugin with the below Krew command:

kubectl krew install kubectl-login

Your OpenID Connect provider must have the default callback endpoint for the Kubernetes API Client listed within the OpenID configuration:

http://localhost:33768/auth/callback

This Kubectl plugin takes the OpenID Connect (OIDC) issuer URL from your .kube/config, so it must be placed in your .kube/config. Once you have made this change to the kubeconfig file, you can proceed to use your username assigned to your OIDC provider:

kubectl login nigeldouglas-oidc

After this command is executed in your CLI, the browser will be opened with a redirect to the OpenID Connect Provider login page. The tokens in your kubeconfig file will be replaced after a successful authentication on the OIDC provider’s end.

7. Kubectl-whisper-secret

Link to GitHub Repository

We mentioned the importance of securing sensitive credentials like ‘Secrets’ using the kubectl-ssm-secret plugin. The whisper-secret plugin focuses on creating those secrets with improved privacy. The plugin allows users to create secrets with secure input prompts to prevent information leakages through terminal (bash) history, shoulder surfing attacks, etc.

You can install the whisper-secret plugin with the below Krew command:

kubectl krew install whisper-secret

’kubectl create secret’ has a few sub-commands we use most often that can possibly leak sensitive information in multiple ways, as mentioned above. For example, you can connect to a Docker registry via the ’kubectl create secret’ command with a plain-text password for authentication.

kubectl create secret docker-registry my-secret --docker-password nigelDouglasP@ssw0rD

’kubectl whisper-secret’ plugin allows users to create secrets with a secure input prompt for fields like --from-literal and --docker-password that contain sensitive information.

kubectl whisper-secret docker-registry my-secret --docker-password -- -n nigel-test --docker-username <insert-password>

You are then prompted to enter the Docker password, but this is not inserted into the command itself. This way, the password will not show-up in the bash history as a plain text value, increasing security.

8. Kubectl-capture

Link to GitHub Repository

Sysdig open source (Sysdig Inspect) is a powerful tool for container troubleshooting, performance tuning, and security investigation. The team at Sysdig created a kubectl plugin which triggers a packet capture in the underlying host which is running a pod.

You can install the kubectl-capture plugin with the below Krew command:

kubectl krew install kubectl-capture

Packet captures are incredibly useful for incident response and forensics in Kubernetes. The capture file is created for a duration of time and is downloaded locally in order to use it with Sysdig Inspect, a powerful open source interface designed to intuitively navigate the data-dense Sysdig captures that contain granular system, network, and application activity of a Linux system.

Simply run the below command against any running pod in the cluster:

kubectl capture kinsing-78f5d695bd-bcbd8

When the capture container is being spun, it takes some time to compile the Sysdig Kernel module and capture system calls. Once completed, you can read the content within the Sysdig Inspect UI from your workstation:

With these tools, it will be much easier for the analyst to find the source of the problem or to audit what happened. If you want to go deeper, you can read container troubleshooting with Sysdig Inspect or triaging malicious containers.

9. Kubectl-trace

Link to GitHub Repository

kubectl-trace is a kubectl plugin that allows you to schedule the execution of bpftrace programs in your Kubernetes cluster. In short, Kubectl-trace plugin is a tool for distributed tracing in Kubernetes clusters. It allows you to trace the execution of requests as they pass through different components of a cluster, including pods, services, and ingress controllers.

You can install the kubectl-trace plugin with the below Krew command:

kubectl krew install trace

One potential security benefit of using the Kubectl-trace plugin is that it can help you identify and troubleshoot issues related to request handling within a cluster. For example, if you suspect that a particular request is being blocked or slowed down due to some issue in the cluster, you can use Kubectl-trace to track the request as it travels through the cluster and identify the source of the problem.

This plugin runs a program that probes a tracepoint on the node of choice:

kubectl trace run <node-name> -e "tracepoint:syscalls:sys_enter_* { @[probe] = count(); }"

Another potential security benefit is that Kubectl-trace can help you understand how requests are being handled within a cluster, which can be useful for identifying potential vulnerabilities or misconfigurations. For example, if you see that a request is being handled by a pod or service that has been compromised, you can use Kubectl-trace to track the request and identify the source of the issue.

Overall, the Kubectl-trace plugin can be a useful tool for improving the security of a Kubernetes cluster by helping to identify and address issues related to request handling and execution.

10. Access-matrix

Link to GitHub Repository

Access-matrix (often referred to as ‘Rakkess’) is a kubectl plugin that shows an access matrix for your server resources.

You can install the access-matrix plugin with the below Krew command:

kubectl krew install access-matrix

Simply run the below command to see the Create, Read, Update & Delete (CRUD) permissions for all resources in the ‘default’ network namespace:

kubectl rakkess –n default

Some roles only apply to resources with a specific name. To review such configurations, provide the resource name as an additional argument. For example, show access rights for the ConfigMap called sysdig-controller in namespace sysdig-agent:

kubectl access-matrix r cm sysdig-controller -n sysdig-agent --verbs=all

As rakkess resource needs to query Roles, ClusterRoles, and their bindings, it usually requires administrative cluster access.

11. Rolesum

Link to GitHub Repository

The Rolesum kubectl plugin is a tool for generating a summary of the roles and permissions defined in a Kubernetes cluster. It allows you to see all of the roles and permissions that have been defined in a cluster, along with the users and groups that have been granted those roles. Summarize RBAC roles for the specified subject (ServiceAccount, User, and Group).

You can install the rolesum plugin with the below Krew command:

kubectl krew install rolesum

One potential security benefit of using the Rolesum kubectl plugin is that it can help you identify and understand the roles and permissions that have been defined in a cluster. This can be useful for ensuring that appropriate access controls have been put in place, and for identifying potential vulnerabilities or misconfigurations.

You can summarize roles bound to the "nigeldouglas" ServiceAccount.

By default, rolesum looks for serviceaccounts. There’s no need to specify any flag.

kubectl rolesum nigeldouglas

Another potential security benefit is that Rolesum can help you quickly identify users and groups that have been granted certain roles or permissions, which can be useful for troubleshooting issues or for performing security assessments.

For example, you can summarize roles bound to the "staging" group.

kubectl rolesum -k Group staging

Overall, the Rolesum kubectl plugin can be a useful tool for improving the security of a Kubernetes cluster by helping you understand and manage the roles and permissions that have been defined in the cluster.

12. Cert-Manager

Link to GitHub Repository

Cert-Manager is a Kubectl plugin that provides automatic management of Transport Layer Security (TLS) certificates within a cluster. It allows you to easily provision, manage, and renew TLS certificates for your applications without having to manually handle the certificate signing process.

You can install the cert-manager plugin with the below Krew command:

kubectl krew install cert-manager

One potential security benefit of using cert-manager is that it can help you ensure that your applications are using valid, up-to-date TLS certificates. This can be important for protecting the confidentiality and integrity of communication between your applications and their users.

Another potential security benefit is that cert-manager can help you automate the process of obtaining and renewing TLS certificates, which can reduce the risk of certificate expiration or mismanagement.

Overall, the cert-manager kubectl plugin can be a useful tool for improving the security of a Kubernetes cluster by helping you to manage TLS certificates in a secure and automated manner. The cert-manager plugin is loosely based upon the work of kube-lego and has borrowed some wisdom from other similar projects, such as kube-cert-manager.

13. np-viewer

Link to GitHub Repository

The kubectl-np-viewer plugin is a tool for visualizing the network topology of a Kubernetes cluster. It allows you to view the connections between pods, services, and other resources within a cluster in a graphical format.

You can install the np-viewer plugin with the below Krew command:

kubectl krew install np-viewer

Unlike the Cilium plugin we mentioned earlier, the kubectl-np-viewer plugin helps users understand and visualize the communication patterns within a cluster regardless of the CNI plugin used. The Cilium plugin only helps manage Cilium resources, such as the Cilium network policy. By viewing the default Kubernetes network policies, teams who are starting off with Kubernetes networking benefit from useful visibility into potential vulnerabilities or misconfigurations, such as pods that are communicating with unintended resources or are exposed to the internet.

The below command prints network policies rules affecting a specific pod in the current namespace:

kubectl np-viewer -p pod-name

Similarly, a potential security benefit from the kubectl-np-viewer plugin is that it helps users troubleshoot network issues within a cluster. For example, if you are experiencing connectivity issues between pods or services, you can use the plugin to visualize the connections between those resources and identify the source of the problem across all network namespace.

The below command prints all network policies rules for all namespaces:

kubectl np-viewer --all-namespaces

Overall, the kubectl-np-viewer plugin can be a useful tool for improving the security of a Kubernetes cluster by helping you to understand and monitor the network topology of the cluster. Not all businesses have moved to advanced network policy implementations, such as Calico and Cilium. While users are exploring the Kubernetes Network Policy implementation, they can better understand how their policies control potentially unwanted/malicious traffic within their cluster with this security plugin.

14. ksniff

Link to GitHub Repository

The ksniff kubectl plugin is a tool for capturing and analyzing network traffic in a Kubernetes cluster. It can be used to troubleshoot network issues, monitor traffic patterns, and perform security assessments.

You can install the ksniff plugin with the below Krew command:

kubectl krew install ksniff

One benefit of using ksniff is that it allows you to capture and analyze traffic without having to directly access the nodes in a cluster. This can be helpful in situations where you don't have direct access to the nodes, or where you want to minimize the potential impact of capturing traffic on the cluster.

Another benefit is that ksniff can be used to capture traffic between pods and services, which can be useful for understanding how applications communicate within a cluster. This is helpful for troubleshooting issues, optimizing performance, and identifying potential security vulnerabilities.

Overall, the ksniff kubectl plugin can be a useful tool for improving the security of a Kubernetes cluster by helping to identify and address network-related issues and vulnerabilities. It achieves this by sniffing on Kubernetes pods with existing technologies, such as TCPdump and WireShark.

15. Inspektor-Gadget

Link to GitHub Repository

Inspektor-Gadget is one of the most useful kubectl plugins. The plugin executes within the user's system and as a DaemonSet when deployed within the cluster. It is actually a collection of tools (or gadgets) to debug and inspect Kubernetes resources and applications.

You can install the gadget plugin with the belowKkrew command:

kubectl krew install gadget

You can deploy one or more gadgets. Example gadgets are categorized into:

Advice (Generates seccomp profiles and network policies for the cluster)
Audit (Traces the system calls that the seccomp profile sends to the audit log)
Profile (Analyzes Block I/O through distributed latency and CPU Perf by sampled stack traces)
Snapshot (Gather information about running processes and TCP/UDP sockets)
Top (Periodically report block device I/O activity, eBPF runtime stats, and read/write activity by file)
Trace (Trace almost all activity from DNS queries/responses to OOMkill triggering a process kill)

It manages the packaging, deployment, and execution of eBPF programs in a Kubernetes cluster, including many based on BPF Compiler Collection (BCC) tools, as well as some developed specifically for use in Inspektor Gadget. It automatically maps low-level kernel primitives to high-level Kubernetes resources, making it easier and quicker to find the relevant information.

To “Advise” on a Kubernetes Network Policy based on network trace activity, run the below command:

kubectl gadget advise network-policy report --input ./networktrace.log > network-policy.yaml

To “Audit” a seccomp profile based on pods, namespaces, syscalls, and code, run the below command:

kubectl gadget audit seccomp -o custom-columns=namespace,pod,syscall,code

DIY kubectl plugins

You can write a plugin in any programming language or script that allows you to write command-line commands. There is no plugin installation or pre-loading required, which makes compiling these plugins rather simple.

Plugin executables receive the inherited environment from the kubectl binary. The plugin will then determine which command path it wishes to implement based on the name – for example, a plugin named kubectl-sysdig provides a command kubectl sysdig.

You must install the plugin executable somewhere in your PATH.

A plugin script would look something like this:

#!/bin/bash
# optional argument handling
if [[ "$1" == "version" ]]
then
    echo "1.0.0"
    exit 0
fi
# optional argument handling
if [[ "$1" == "config" ]]
then
    echo "$KUBECONFIG"
    exit 0
fi
echo "I am a plugin named kubectl-sysdig"

For a complete guide on building Kubectl plugins, check out the official Kubernetes documentation.

Final considerations on kubectl plugins

At the time of writing this blog post, there were 208* kubectl plugins currently accessible on Krew. Those kubectl plugins are accessible to developers across all major platforms, like MacOS, Linux, and Windows. While these plugins often address clear limitations over the default kubectl utility for operational tasks and security auditing, they also open a bunch of new security gaps for your Kubernetes cluster.

From a security standpoint, we discussed 15 of the most useful kubectl plugins for giving security teams better visibility for incident response and forensics in Kubernetes. However, as we add more plugins into the environment, we are also adding additional un-audited binaries that could be compromised. Krew does not provide an obligation to audit these binaries for known vulnerabilities or insecure configurations.

Some security implications of using kubectl plugins include:

Plugin vulnerabilities: If a kubectl plugin has a vulnerability, it can potentially be exploited by an attacker to gain access to your Kubernetes cluster.
Insecure plugin installation: If a plugin is installed from an untrusted source, it could contain malicious code that could compromise the security of your cluster.
Privilege escalation: kubectl plugins run with the same privileges as the kubectl command, so if a plugin is compromised, it could potentially be used to escalate privileges and gain access to sensitive resources in your cluster.
Data leakage: If a kubectl plugin is not properly secured, it could potentially leak sensitive data from your cluster.

To mitigate these risks, it is important to only install kubectl plugins from trusted sources and to regularly update and patch any plugins you have installed. It is also a good idea to regularly review the plugins you have installed and remove any that are no longer needed.

If you don’t feel like a specific plugin adds value to your cluster, it would be wise to remove it just in case.

How attackers use exposed Prometheus server to exploit Kubernetes clusters

Miguel — Fri, 02 Dec 2022 11:53:43 +0000

What is the main thing we want to explain in this article? It’s simple; don’t expose your metrics for free.

Sometimes we think about deep and complex defense methods and that’s fine. We don’t know why, but we always forget about the base. Don’t expose your data. By default, your Prometheus server can allow anyone to make queries to get information from your Kubernetes Cluster.

This is not something new. In 2018, Tesla had a cryptocurrency mining application in their cloud account, and the initial access was an exposed Kubernetes Dashboard with credentials in the clear.

Moreover, we are not the first to talk about (in) security in monitoring tools. Here are three good examples:

With this in mind, are exposed Prometheus servers a real attack surface?

Prometheus exposed in the wild

One of the most important steps in any pentest, ethical hacking, or real attack is gathering as much information you can get from the target.

The fastest way to check if something is exposed on the internet is to query Google. The specific queries to gather information are denominated Google Dorking and, in our case, is something trivial to get real exposed Prometheus.

A cooler way to find exposed Prometheus servers is using search engines. We used the most common ones to check how many servers we could access:

Search Engine	Number exposed Prometheus server
Shodan	31,679
Censys	61,854
Fofa	161,274

At this point, we would like to clarify a critical fact.

Disclaimer: We have not used an actual exposed Prometheus server to consult or prepare for this talk. We performed all testing in our demo environment and strongly recommend always following security best practices.

After that, what can we do if we have access to a Prometheus server and have access to the fingerprint Kubernetes?

Prometheus’ exporters and fingerprint Kubernetes

Prometheus is the de facto monitoring standard in Kubernetes. All the Kubernetes components of the control plane generate Prometheus metrics out of the box, and many Kubernetes distributions come with Prometheus installed by default including a series of standard exporters, generally:

Node Exporter for infrastructure and host metrics.
KSM Exporter for Kubernetes objects state metrics.

An exporter is an application that generates metrics from other applications or systems that do not expose Prometheus metrics natively.

Cloud provider, where are you?

Imagine that you have a possible target in www.example.com.

All you know is that this site is a web page with users and a little e-commerce section. Under that domain, you find an open exposed Prometheus. The first thing you can do is try to identify the cloud provider where the site is hosted.

You can use the metric node_dmi_info from the Node Exporter. This metric is very interesting, as it gives information about each Kubernetes node:

System vendor: It exposes the cloud vendor’s name. Some example values could be “Amazon EC2” or “Tencent Cloud.”
Product name: Useful to identify both the cloud provider and the product used, as we can find some popular product names from the AWS EC2 catalog (like “m5.xlarge“) or other vendors’ products.

But the cloud provider, even if interesting, is still so vague. You can gather more information if you focus on networking. You can start with the node_network_info metric from the Node Exporter. And even more, you can narrow your search if you filter only the Ethernet interfaces.

Why only Ethernet ones? Because usually, they are the ones that the host identifies as physical network connections and are used to connect the host with the outside world and other machines.

node_network_info{device=~'eth.+'}

This query provides the following information:

IP address of each host.
Device ID.
Availability zone of the cloud provider.
ID of the VPC (Virtual Private Cloud).

Here is an example of some possible values:

    address="06:d5:XX:XX:XX:XX"
    broadcast="ff:ff:ff:ff:ff:ff"
    device="eth0"
    instance="172.31.XX.XX:9100"
    instance_az="us-west-2a"
    instance_id="i-XXXXX"
    instance_name="XXX-XXX"
    instance_type="c5.xlarge"
    instance_vpc="vpc-XXXXXXX"
    operstate="up"

You can also get more information, like the hostname of each node with the metric kube_node_info from KSM.

The long and windy road to the pod

This was all about physical info, but how can we get from outside the web page to a pod in the cluster? The answer to this question is in the ingress and the services.

The ingress controllers in Kubernetes act as reverse proxies and allow redirecting different paths of the URL to different Kubernetes services. These services normally act as load balancers in front of a set of pods that expose a port for connections. The metric kube_ingress_path from KSM will give you information about the URL paths and the associated services of the ingress controllers in your cluster.

This way, you can know that the path /api/users/login goes to the Kubernetes service users-login of the namespace api. Funny, right?

Load balancer services are a special kind of Kubernetes service. Cloud providers use those load balancer services to expose the service to the outside world. As an example, when you create a load-balancer service in an AWS Kubernetes cluster, it creates an ELB (Elastic Load Balancer) instance bound to the service.

This promQL query will give you information about all the load-balancer services in a Kubernetes cluster:

kube_service_info * on (service) group_left group by (service,type) (kube_service_spec_type{type="LoadBalancer"})

To guess what pods are behind each service, you have two options. You can check the metric kube_pod_labels from KSM. These labels are the ones that the service usually uses to select the pods that will serve the requests, but unfortunately, there is not a direct way to get the association between pods and service in pure KSM.

However, if you are lucky enough, the cluster will have installed the OpenCost exporter, a tool that helps infrastructure engineers understand the costs of their cloud usage. This exporter generates an interesting metric called service_selector_labels, which directly gives you the association between the service and the labels that the pod needs to have to be part of that particular service.

This promQL query will give you the labels of each workload used for matching in services:

avg by (namespace,label_app,owner_name)(kube_pod_labels{app="cost-model"} * on(namespace,pod) group_left(owner_name) kube_pod_owner{job="kube-state-metrics"})

While this other one will give you the labels that each service uses to find the pods:

avg by (namespace,label_app, service)(service_selector_labels)

Being a many-to-many association, there is not an easy way to collect all this info in a single promQL query, but the info is there, and it’s easy to make a quick correlation between services and pods.

This way, we have all the points of the path from the URL to the pods: the path of the URL (thanks to the ingress), and pods serving the requests (thanks to the services and labels of the pods).

Logical song of the cluster

You used the metric kube_node_info to get information on the nodes, but now, you are also interested in making a logical map of namespaces, workloads, and pods inside the Kubernetes cluster.

This is easy by using the KSM metrics. The metric kube_namespace_status_phase gives you all the namespaces in the cluster. From there, you can go down with the following metrics for each of the different workload types:

kube_deployment_spec_replicas
kube_daemonset_status_desired_number_scheduled
kube_statefulset_replicas
kube_replicaset_spec_replicas
kube_cronjob_info

After that, you can get info on the pods using kube_pod_info, and associating them with their workloads with kube_pod_owner in the following promQL:

kube_pod_info * on(namespace,pod) group_left(owner_name) kube_pod_owner

Finally, you can even get the container inside each pod with the metric kube_pod_container_info. For example, a pod called postgres-db can have two containers named postgresql and postgres-exporter.

But there is more. You can not only know the namespace and workload of a pod, you can also discover the node where it is living thanks to the label node of the metric kube_pod_info. Why is this important? Keep reading.

The boulevard of broken nodes

You used the metric kube_node_info before to get the hostname of each node, but this metric has more surprises to unfold.

Two labels of this metric will give us full information about the Operative System image used to build the node and the detailed kernel version.

os_image
kernel_version

A quick search on CVE for “Ubuntu 18.04.4 LTS” or “Linux 3.10.0-1160.59.1.el7.x86_64” will give a possible attacker a good set of exploits to use if they can get access to the machine.

Let’s talk about K8s

You have done a good job gathering information about the cluster so far. Namespaces, pods, services, and more But what about Kubernetes itself? There is a set of processes in Kubernetes itself that are just there, and we don’t even think about them unless they start causing problems. We are talking about the Kubernetes control plane.

What would you say if we tell you that there is a metric that specifies the specific version of each of the components of the control plane? While presenting Prometheus, we said that Kubernetes control plane components were exposing natively metrics. Well, one of those metrics is kubernetes_build_info. This gives you information about, not only the full (major and minor) version of each component, but also the git commit and the build date.

This is great if you want to know if a concrete vulnerability affects one of the control plane components of the cluster (among other things).

We have a secret…

Everybody loves secrets, especially attackers. In KSM, there is a metric called kube_secret_info that gives you information about the namespace, node, and name of the secrets of your cluster.

But if you are interested in knowing the content of the secrets, you can use this query:

kube_secret_annotations{kubectl_kubernetes_io_last_applied_configuration != ""}

Why? Well, this is somehow embarrassing. In some older versions of kubectl, it used to save the last applied configuration in an annotation. This was being made for every object, including secrets. This had the effect that, even if the secret was only accessible by the service accounts and role bindings that you can imagine, Prometheus can expose the content of the secret in plain text in that metric.

On images and registries

Do you think you had enough? There is one more interesting thing you can get from KSM. The metric kube_pod_container_info has an interesting piece of information in these labels:

image: name and tag of the image of the container (for example docker.io/library/cassandra:3.11.6)
image_id: name, tag, and hash of the image of the container

This gives you information about:

Application used.
Registry used to pull the image.
Image used.
Tag of the image.
Hash that identifies uniquely the image independently of its tag.

Summary Kubernetes fingerprint

Let’s see what you’ve done so far. You gathered information about:

Cloud provider.
Kubernetes control plane components versions.
Network path from the outside to pods.
Nodes hostnames and IPs.
Operative system and kernel versions.
Logical structure of the cluster namespaces, workloads, and pods.
Images used for the containers, from the source repository to the image tag.
Annotations and names of the secrets of the cluster.

All this information is enough to make a good surface attack analysis of the cluster.

Ninja mode!

Do you want to hear something funny? We gathered all this information and most likely, there is not a trace of all the queries that we did to get it. Prometheus can register logs of the queries, but that’s disabled by default. You can even check if your activity is being logged with this metric:

prometheus_engine_query_log_enabled

Inside the attackers’ minds

Now, attackers just need to know what their target is. In 99% of attacks, it’s money, but how to get the money from the victim, that’s the point where the attacker’s path is defined.

In the talk, we exposed three examples and in each of them, the tools and services exploited are different. The important thing is that we already know where the weaknesses are.

Leak sensitive data

In the first scenario, the exposed application is running on a Kubernetes cluster and the attacker wants to access the data without authorization. The first thing the attacker could check is if the application can be exploited through normal pentesting techniques, for example, with SQLmap the attacker can try to gain access to the data.

But if this does not work, what is the next step?

The attacker can check if the container has vulnerable dependencies or if the image used could be exploited, then see if the components or the node itself are exploitable. But everything seems to be fine. There are no CVE matches and no known exploits that could be used to gain initial access.

What’s next? Well, Prometheus exposed the image and registry that the attacker accessed, but what about attacking the supply chain? In this case, we have two scenarios:

Official/private registration: In this case, the attacker could use similar image names, such as homographs, visually similar by using different Unicode groups, to trick the target. Another technique could be to abuse an insider to manually change the exposed image. In this case, it depends on the financial gain of the attacker.
Third-party registry: In this case, one of the methods could be social engineering, using tools like BeeF to create a specific phishing or fake page to get the login credentials and change the image to a new one with a known and exploitable vulnerability and wait for the deployment. One more thing is this is not magic or 100% successful. If the company scans the images in the deployment, it could be detected!

Cryptomining

In this scenario, one of the most relevant in the last years with the era of cloud, the attacker would like to get access to the cloud account where the application or Kubernetes cluster are deployed. The attacker could take two paths. The long path was to identify one app exposed via Ingress-controller that has a known vulnerability easily exploited via HTTP and obtain a Remote Code Execution inside the container.

The vulnerability exploited in this case will be the infamous log4shell.

Once the attacker has access to the container, they don’t even need to gather more information about the cluster or the node because Prometheus exposed this information as well. From that, we could directly exploit another vulnerability to escape to the container and get full access to the node without using more tools or scanning, evading typical defense methods.

Note: This is not 100% successful. If runtime security is used and the shell within the container is detected as malicious behavior, the incident will be detected before impacting resources.

Now that the attacker has full control of the node, they will be able to deploy containers to run cryptominers, or find cloud credentials in configuration files or env variables to gain initial access.

But this is a long way, what’s the short way? Well, it is possible for Prometheus to directly expose credentials to these cloud providers in the same way that the Kubernetes Dashboard did in the past. Therefore, the attacker only needs to query the information via query and get the API keys in clear text.

Ransomware

Yes, ransomware in Kubernetes is not typical but not impossible. The scenario is similar to the previous one. We need to get write access and for that, we need to jump or move between namespaces.

In this case, we find another application with a different vulnerability, Spring Cloud, but with the same purpose: to get a shell inside the container.

Once inside, we know that a Kubernetes component is an old vulnerable version that we can exploit to get access to etcd, and with that, full access to the namespaces.

The curious thing here is after the data is encrypted, the attacker needs to ask for a ransom through some channel. In a typical scenario, our PC would be locked and the screen would show instructions to pay via BTC or ETH, but inside the container. We hate to share ideas with the bad guys, but one option could be to deploy a container with a modified UI and force ingress to display this in front of the actual application.

Conclusion

We might think that metrics are not important from a security perspective, but we demonstrated that’s not true. Kubernetes and Prometheus advise problems with exposing your data to the world, but regardless, these problems are still widespread.

Following the security best practices in every part of our chain leads to being safe from most security incidents. Otherwise, we will change the typical scenario with a long battle between attackers and defenders for a speedrun.

We will have to continue to fight with new vulnerabilities that impact our services and also a plan against insiders. But let’s at least make things difficult for them.

If you want to see the talk:

The slides are available here.

26 AWS Security Best Practices to Adopt in Production

Miguel — Wed, 21 Sep 2022 08:42:58 +0000

One of the most important pillars of a well-architected framework is security. Thus, it is important to follow these AWS security best practices to prevent unnecessary security situations.

So, you’ve got a problem to solve and turned to AWS to build and host your solution. You create your account and now you’re all set up to brew some coffee and sit down at your workstation to architect, code, build, and deploy. Except, you aren’t.

There are many things you must set up if you want your solution to be operative, secure, reliable, performant, and cost effective. And, first things first, the best time to do that is now – right from the beginning, before you start to design and engineer.

Initial AWS setup

Never, ever, use your root account for everyday use. Instead, head to Identity and Access Management (IAM) and create an administrator user. Protect and lock your root credentials in a secure place (is your password strong enough?) and, if your root user has keys generated, now is the best time to delete them.

You will absolutely want to activate Multi Factor Authentication (MFA) too for your root account. You must end up with a root user with MFA and no access keys. And you won’t use this user unless strictly necessary.

Now, about your newly created admin account, activating MFA for it is a must. It’s actually a requirement for every user in your account if you want to have a security first mindset (and you actually want to), but especially so for power users. You will only use this account for administrative purposes.

For daily use, you need to go to the IAM panel and create users, groups, and roles which can access only the resources to which you explicitly grant permissions.

Now you have:

Root account (with no keys) securely locked into a safe.
Admin account for administrative use.
Several users, groups, and roles for day to day use.

All of them should have MFA activated and strong passwords.

You are almost ready to begin your actual work, but first, a word of caution about the AWS shared responsibility model.

Security and compliance is a shared responsibility between AWS and the customer. AWS operates, manages, and controls the components from the host operating system and virtualization layer, down to the physical security of the facilities in which the service operates. The customer assumes responsibility and management of the guest operating system (including updates and security patches), other associated application software, as well as the configuration of the AWS provided security group firewall.

Therefore, the management and application of diligent AWS security is the responsibility of the customer.

AWS Cloud security best practices checklist

In this section we will walk through the most common AWS services and provide 26 security best practices to adopt.

AWS Security with open source – Cloud Custodian is a Cloud Security Posture Management (CSPM) tool. CSPM tools evaluate your cloud configuration and identify common configuration mistakes. They also monitor cloud logs to detect threats and configuration changes.

Now let’s walk through service by service.

AWS security best practices by service
	High Risk 🟥🟥🟥	Medium Risk 🟨🟨	Low Risk 🟩
AWS IAM	(1) IAM policies should not allow full “*” administrative privileges (4) IAM root user access key should not exist (6) Hardware MFA should be enabled for the root user	(3) IAM users’ access keys should be rotated every 90 days or less (5) MFA should be enabled for all IAM users that have a console password (7) Password policies for IAM users should have strong configurations (8) Unused IAM user credentials should be removed	(2) IAM users should not have IAM policies attached
Amazon S3	(10) S3 buckets should have server-side encryption enabled	(9) S3 Block Public Access setting should be enabled (11) S3 Block Public Access setting should be enabled at the bucket level
AWS CloudTrail	(12) CloudTrail should be enabled and configured with at least one multi-Region trail	(13) CloudTrail should have encryption at rest enabled (14) Ensure CloudTrail log file validation is enabled
AWS Config	(15) AWS Config should be enabled
Amazon EC2	(16) Attached EBS volumes should be encrypted at rest (19) EBS default encryption should be enabled		(17) VPC flow logging should be enabled in all VPCs (18) The VPC default security group should not allow inbound and outbound traffic
AWS DMS	(20) AWS Database Migration Service replication instances should not be public
Amazon EBS	(21) Amazon EBS snapshots should not be public, determined by the ability to be restorable by anyone
Amazon OpenSearch Service	(22) Elasticsearch domains should have encryption at rest enabled
Amazon SageMaker		(23) SageMaker notebook instances should not have direct internet access
AWS Lambda		(24) Lambda functions should use supported runtimes
AWS KMS		(25) AWS KMS keys should not be unintentionally deleted
Amazon GuardDuty		(26) GuardDuty should be enabled

AWS Identity and Access Management (IAM)

AWS Identity and Access Management (IAM) helps enforce least privilege access control to AWS resources. You can use IAM to restrict who is authenticated (signed in) and authorized (has permissions) to use resources.

1.- Do not allow full “*” administrative privileges on IAM policies 🟥🟥🟥

IAM policies define a set of privileges that are granted to users, groups, or roles. Following standard security advice, you should grant least privilege, which means to allow only the permissions that are required to perform a task.

When you provide full administrative privileges instead of the minimum set of permissions that the user needs, you expose the resources to potentially unwanted actions.

For each AWS account, list the customer managed policies available:

aws iam list-policies --scope Local --query 'Policies[*].Arn'

The previous command will return a list of policies along with their Amazon Resource Names (ARNs). Using these ARNs, now retrieve the policy document in JSON format:

aws iam get-policy-version
--policy-arn POLICY_ARN
--version-id v1
--query 'PolicyVersion.Document'

The output should be the requested IAM policy document:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "1234567890",
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Look into this document for the following elements:

"Effect": "Allow", "Action": "*", "Resource": "*"

If these elements are present, then the customer-managed policy allows full administrative privileges. This is a risk and must be avoided, so you will need to tune these policies down to pinpoint exactly what actions you want to allow for each specific resource.

Repeat the previous procedure for the other IAM customer managed policies.

If you want to detect the use of full administrative privileges with open source, here is a Cloud Custodian rule:

- name: full-administrative-privileges
  description: IAM policies are the means by which privileges are granted to users, groups, or roles. It is recommended and considered a standard security advice to grant least privilege -that is, granting only the permissions required to perform a task. Determine what users need to do and then craft policies for them that let the users perform only those tasks, instead of allowing full administrative privileges.
  resource: iam-policy
  filters:
    - type: used
    - type: has-allow-all

2.- Do not attach IAM policies to users 🟩

By default, IAM users, groups, and roles have no access to AWS resources.

IAM policies grant privileges to users, groups, or roles. We recommend that you apply IAM policies directly to groups and roles but not to users. Assigning privileges at the group or role level reduces the complexity of access management as the number of users grows. Reducing access management complexity might in turn reduce the opportunity for a principal to inadvertently receive or retain excessive privileges.

3.- Rotate IAM users’ access keys every 90 days or less 🟨🟨

AWS recommends that you rotate the access keys every 90 days. Rotating access keys reduces the chance that an access key that is associated with a compromised or terminated account is used. It also ensures that data cannot be accessed with an old key that might have been lost, cracked, or stolen. Always update your applications after you rotate access keys.

First, list all IAM users available in your AWS account with:

aws iam list-users --query 'Users[*].UserName'

For all the users returned by this command, determine each active access key lifetime by doing:

aws iam list-access-keys --user-name USER_NAME

This should expose the metadata for each access key existing for the specified IAM user. The output will look like this:

{
    "AccessKeyMetadata": [
        {
            "UserName": "some-user",
            "Status": "Inactive",
            "CreateDate": "2022-05-18T13:43:23Z",
            "AccessKeyId": "AAAABBBBCCCCDDDDEEEE"
        },
        {
            "UserName": "some-user",
            "Status": "Active",
            "CreateDate": "2022-03-21T09:12:32Z",
            "AccessKeyId": "AAAABBBBCCCCDDDDEEEE"
        }
    ]
}

Check the CreateDate parameter value for each active key to determine its creation time. If an active access key has been created before the last 90 days, the key is outdated and must be rotated to secure the access to your AWS resources.

Repeat for each IAM user existing in your AWS account.

4.- Ensure that IAM root user access keys do not exist 🟥🟥🟥

As we stated during your initial setup, we highly recommend that you remove all access keys that are associated with the root user. This limits the vectors that can be used to compromise your account. It also encourages the creation and use of role-based accounts that are least privileged.

The following Cloud Custodian rule will check if root access keys have been used on your account:

- name: root-access-keys
  description: The root user account is the most privileged user in an AWS account. AWS Access Keys provide programmatic access to a given AWS account. It is recommended that all access keys associated with the root user account be removed.
  resource: account
  filters:
    - type: iam-summary
      key: AccountAccessKeysPresent
      value: 0
      op: gt

5.- Enable MFA for all IAM users that have a console password 🟨🟨

Multi-factor authentication (MFA) adds an extra layer of protection on top of a username and password. With MFA enabled, when a user signs in to an AWS website, they are prompted for their username and password. In addition, they are prompted for an authentication code from their AWS MFA device.

We recommend that you enable MFA for all accounts that have a console password. MFA is designed to provide increased security for console access. The authenticating principal must possess a device that emits a time-sensitive key and must have knowledge of a credential.

6.- Enable hardware MFA for the root user 🟥🟥🟥

Virtual MFA might not provide the same level of security as hardware MFA devices. A hardware MFA has a minimal attack surface, and cannot be stolen unless the malicious user gains physical access to the hardware device. We recommend that you use only a virtual MFA device while you wait for hardware purchase approval or for your hardware to arrive, especially for root users.

To learn more, see Enabling a virtual multi-factor authentication (MFA) device (console) in the IAM User Guide.

Here is a Cloud Custodian rule to detect lack of root hardware MFA:

- name: root-hardware-mfa
  description: The root user account is the most privileged user in an AWS account. MFA adds an extra layer of protection on top of a username and password. With MFA enabled, when a user signs in to an AWS website, they will be prompted for their username and password as well as for an authentication code from their AWS MFA device. It is recommended that the root user account be protected with a hardware MFA.
  resource: account
  filters:
    - or:
      - type: iam-summary
        key: AccountMFAEnabled
        value: 1
        op: ne
      - and:
        - type: iam-summary
          key: AccountMFAEnabled
          value: 1
          op: eq
        - type: has-virtual-mfa
          value: true

7.- Ensure those password policies for IAM users have strong configurations 🟨🟨

We recommend that you enforce the creation of strong user passwords. You can set a password policy on your AWS account to specify complexity requirements and mandatory rotation periods for passwords.

When you create or change a password policy, most of the password policy settings are enforced the next time users change their passwords. Some of the settings are enforced immediately.

What constitutes a strong password is a subjective matter, but the following settings will put you on the right path:

RequireUppercaseCharacters: true
RequireLowercaseCharacters: true
RequireSymbols: true
RequireNumbers: true
MinimumPasswordLength: 8

8.- Remove unused IAM user credentials 🟨🟨

IAM users can access AWS resources using different types of credentials, such as passwords or access keys. We recommend you remove or deactivate all credentials that were unused for 90 days or more to reduce the window of opportunity for credentials associated with a compromised or abandoned account to be used.

You can use the IAM console to get some of the information that you need to monitor accounts for dated credentials. For example, when you view users in your account, there is a column for Access key age, Password age, and Last activity. If the value in any of these columns is greater than 90 days, make the credentials for those users inactive.

You can also use credential reports to monitor user accounts and identify those with no activity for 90 or more days. You can download credential reports in .csv format from the IAM console.

For more information, check out AWS security best practices for IAM in more detail.

Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. There are few AWS security best practices to adopt when it comes to S3.

9.- Enable S3 Block Public Access setting 🟨🟨

Amazon S3 public access block is designed to provide controls across an entire AWS account or at the individual S3 bucket level to ensure that objects never have public access. Public access is granted to buckets and objects through access control lists (ACLs), bucket policies, or both.

Unless you intend to have your S3 buckets be publicly accessible, you should configure the account level Amazon S3 Block Public Access feature.

Get the names of all S3 buckets available in your AWS account:

aws s3api list-buckets --query 'Buckets[*].Name'

For each bucket returned, get its S3 Block Public Access feature configuration:

aws s3api get-public-access-block --bucket BUCKET_NAME

The output for the previous command should be like this:

"PublicAccessBlockConfiguration": {
  "BlockPublicAcls": false,
  "IgnorePublicAcls": false,
  "BlockPublicPolicy": false,
  "RestrictPublicBuckets": false
}

If any of these values is false, then your data privacy is at stake. Use this short command to remediate it:

aws s3api put-public-access-block
--region REGION
--bucket BUCKET_NAME
--public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

10.- Enable server-side encryption on S3 buckets 🟥🟥🟥

For an added layer of security for your sensitive data in S3 buckets, you should configure your buckets with server-side encryption to protect your data at rest. Amazon S3 encrypts each object with a unique key. As an additional safeguard, Amazon S3 encrypts the key itself with a root key that it rotates regularly. Amazon S3 server-side encryption uses one of the strongest block ciphers available to encrypt your data, 256-bit Advanced Encryption Standard (AES-256).

List all existing S3 buckets available in your AWS account:

aws s3api list-buckets --query 'Buckets[*].Name'

Now, use the names of the S3 buckets returned at the previous step as identifiers to retrieve their Default Encryption feature status:

aws s3api get-bucket-encryption --bucket BUCKET_NAME

The command output should return the requested feature configuration details. If the get-bucket-encryption command output returns an error message, the default encryption is not currently enabled, and therefore the selected S3 bucket does not automatically encrypt all objects when stored in Amazon S3.

Repeat this procedure for all your S3 buckets.

11.- Enable S3 Block Public Access setting at the bucket level 🟨🟨

Unless you intend to have your S3 buckets be publicly accessible, which you probably shouldn’t, you should configure the account level Amazon S3 Block Public Access feature.

You can use this Cloud Custodian rule to detect S3 buckets that are publicly accessible:

- name: buckets-public-access-block
  description: Amazon S3 provides Block public access (bucket settings) and Block public access (account settings) to help you manage public access to Amazon S3 resources. By default, S3 buckets and objects are created with public access disabled. However, an IAM principle with sufficient S3 permissions can enable public access at the bucket and/or object level. While enabled, Block public access (bucket settings) prevents an individual bucket, and its contained objects, from becoming publicly accessible. Similarly, Block public access (account settings) prevents all buckets, and contained objects, from becoming publicly accessible across the entire account.
  resource: s3
  filters:
    - or:
      - type: check-public-block
        BlockPublicAcls: false
      - type: check-public-block
        BlockPublicPolicy: false
      - type: check-public-block
        IgnorePublicAcls: false
      - type: check-public-block
        RestrictPublicBuckets: false

AWS CloudTrail

AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.

The following section will help you configure CloudTrail to monitor your infrastructure across all your regions.

12.- Enable and configure CloudTrail with at least one multi-Region trail 🟥🟥🟥

CloudTrail provides a history of AWS API calls for an account, including API calls made from the AWS Management Console, AWS SDKs, and command line tools. The history also includes API calls from higher-level AWS services, such as AWS CloudFormation.

The AWS API call history produced by CloudTrail enables security analysis, resource change tracking, and compliance auditing. Multi-Region trails also provide the following benefits.

A multi-Region trail helps to detect unexpected activity occurring in otherwise unused Regions.
A multi-Region trail ensures that global service event logging is enabled for a trail by default. Global service event logging records events generated by AWS global services.
For a multi-Region trail, management events for all read and write operations ensure that CloudTrail records management operations on all of an AWS account’s resources.

By default, CloudTrail trails that are created using the AWS Management Console are multi-Region trails.

List all trails available in the selected AWS region:

aws cloudtrail describe-trails

The output exposes each AWS CloudTrail trail along with its configuration details. If IsMultiRegionTrail config parameter value is false, the selected trail is not currently enabled for all AWS regions:

{
    "trailList": [
        {
            "IncludeGlobalServiceEvents": true,
            "Name": "ExampleTrail",
            "TrailARN": "arn:aws:cloudtrail:us-east-1:123456789012:trail/ExampleTrail",
            "LogFileValidationEnabled": false,
            "IsMultiRegionTrail": false,
            "S3BucketName": "ExampleLogging",
            "HomeRegion": "us-east-1"
        }
    ]
}

Verify that all of your trails and make sure at least one is multi-Region.

13.- Enable encryption at rest with CloudTrail 🟨🟨

Check whether CloudTrail is configured to use the server-side encryption (SSE) AWS Key Management Service customer master key (CMK) encryption.

The check passes if the KmsKeyId is defined. For an added layer of security for your sensitive CloudTrail log files, you should use server-side encryption with AWS KMS–managed keys (SSE-KMS) for your CloudTrail log files for encryption at rest. Note that by default, the log files delivered by CloudTrail to your buckets are encrypted by Amazon server-side encryption with Amazon S3-managed encryption keys (SSE-S3).

You can check that the logs are encrypted with the following Cloud Custodian rule:

- name: cloudtrail-logs-encrypted-at-rest
  description: AWS CloudTrail is a web service that records AWS API calls for an account and makes those logs available to users and resources in accordance with IAM policies. AWS Key Management Service (KMS) is a managed service that helps create and control the encryption keys used to encrypt account data, and uses Hardware Security Modules (HSMs) to protect the security of encryption keys. CloudTrail logs can be configured to leverage server side encryption (SSE) and KMS customer created master keys (CMK) to further protect CloudTrail logs. It is recommended that CloudTrail be configured to use SSE-KMS.
  resource: cloudtrail
  filters:
    - type: value
      key: KmsKeyId
      value: absent

You can remediate it using the AWS Console like this:

Sign in to the AWS Management Console at https://console.aws.amazon.com/cloudtrail/.
In the left navigation panel, select Trails.
Under the Name column, select the trail name that you need to update.
Click the pencil icon next to the S3 section to edit the trail bucket configuration.
Under S3 bucket* click Advanced.
Select Yes next to Encrypt log files to encrypt your log files with SSE-KMS using a Customer Master Key (CMK).
Select Yes next to Create a new KMS key to create a new CMK and enter a name for it, or otherwise select No to use an existing CMK encryption key available in the region.
Click Save to enable SSE-KMS encryption.

14.- Enable CloudTrail log file validation 🟨🟨

CloudTrail log file validation creates a digitally signed digest file that contains a hash of each log that CloudTrail writes to Amazon S3. You can use these digest files to determine whether a log file was changed, deleted, or unchanged after CloudTrail delivered the log.

It is recommended that you enable file validation on all trails. Log file validation provides additional integrity checks of CloudTrail logs.

To check this in the AWS Console proceed as follows:

Sign in to the AWS Management Console at https://console.aws.amazon.com/cloudtrail/.
In the left navigation panel, select Trails.
Under the Name column, select the trail name that you need to examine.
Under S3 section, check for Enable log file validation status:
Enable log file validation status. If the feature status is set to No, then the selected trail does not have log file integrity validation enabled. If this is the case, fix it:
1. Click the pencil icon next to the S3 section to edit the trail bucket configuration.
2. Under S3 bucket* click Advanced and search for the Enable log file validation configuration status.
3. Select Yes to enable log file validation, and then click Save.

Learn more about security best practices in AWS Cloudtrail.

AWS Config

AWS Config provides a detailed view of the resources associated with your AWS account, including how they are configured, how they are related to one another, and how the configurations and their relationships have changed over time.

15.- Verify AWS Config is enabled 🟥🟥🟥

The AWS Config service performs configuration management of supported AWS resources in your account and delivers log files to you. The recorded information includes the configuration item (AWS resource), relationships between configuration items, and any configuration changes between resources.

It is recommended that you enable AWS Config in all Regions. The AWS configuration item history that AWS Config captures enables security analysis, resource change tracking, and compliance auditing.

Get the status of all configuration recorders and delivery channels created by the Config service in the selected region:

aws configservice --region REGION get-status

The output from the previous command shows the status of all AWS Config delivery channels and configuration recorders available. If AWS Config is not enabled, the list for both configuration recorders and delivery channels are shown empty:

Configuration Recorders:
Delivery Channels:

Or, if the service was previously enabled but is now disabled, the status should be set to OFF:

Configuration Recorders:
name: default
recorder: OFF
Delivery Channels:
name: default
last stream delivery status: NOT_APPLICABLE
last history delivery status: SUCCESS
last snapshot delivery status: SUCCESS

To remediate this, after you enable AWS Config, configure it to record all resources.

Open the AWS Config console at https://console.aws.amazon.com/config/.
Select the Region to configure AWS Config in.
If you haven’t used AWS Config before, see Getting Started in the AWS Config Developer Guide.
Navigate to the Settings page from the menu, and do the following:
1. Choose Edit.
2. Under Resource types to record, select Record all resources supported in this region and Include global resources (e.g., AWS IAM resources).
3. Under Data retention period, choose the default retention period for AWS Config data, or specify a custom retention period.
4. Under AWS Config role, either choose Create AWS Config service-linked role or choose Choose a role from your account and then select the role to use.
5. Under Amazon S3 bucket, specify the bucket to use or create a bucket and optionally include a prefix.
6. Under Amazon SNS topic, select an Amazon SNS topic from your account or create one. For more information about Amazon SNS, see the Amazon Simple Notification Service Getting Started Guide.
Choose Save.

To go deeper, follow the security best practices for AWS Config.

Amazon EC2

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable computing capacity that you use to build and host your software systems. Therefore, EC2 is one of the core services of AWS and it is necessary to know the best security practices and how to secure EC2.

16.- Ensure attached EBS volumes are encrypted at rest 🟥🟥🟥

It is to check whether the EBS volumes that are in an attached state are encrypted. To pass this check, EBS volumes must be in use and encrypted. If the EBS volume is not attached, then it is not subject to this check.

For an added layer of security to your sensitive data in EBS volumes, you should enable EBS encryption at rest. Amazon EBS encryption offers a straightforward encryption solution for your EBS resources that doesn’t require you to build, maintain, and secure your own key management infrastructure. It uses KMS keys when creating encrypted volumes and snapshots.

Run the describe-volumes command to determine if your EC2 Elastic Block Store volume is encrypted:

aws ec2 describe-volumes
--filters Name=attachment.instance-id, Values=INSTANCE_ID

The command output should reveal the instance EBS volume encryption status (true for enabled, false for disabled).

There is no direct way to encrypt an existing unencrypted volume or snapshot. You can only encrypt a new volume or snapshot when you create it.

If you enable encryption by default, Amazon EBS encrypts the resulting new volume or snapshot by using your default key for Amazon EBS encryption. Even if you have not enabled encryption by default, you can enable encryption when you create an individual volume or snapshot. In both cases, you can override the default key for Amazon EBS encryption and choose a symmetric customer managed key.

17.- Enable VPC flow logging in all VPCs 🟩

With the VPC Flow Logs feature, you can capture information about the IP address traffic going to and from network interfaces in your VPC. After you create a flow log, you can view and retrieve its data in CloudWatch Logs. To reduce cost, you can also send your flow logs to Amazon S3.

It is recommended that you enable flow logging for packet rejects for VPCs. Flow logs provide visibility into network traffic that traverses the VPC and can detect anomalous traffic or provide insight during security workflows. By default, the record includes values for the different components of the IP address flow, including the source, destination, and protocol.

- name: flow-logs-enabled
  description: VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. After you've created a flow log, you can view and retrieve its data in Amazon CloudWatch Logs. It is recommended that VPC Flow Logs be enabled for packet 'Rejects' for VPCs.
  resource: vpc
  filters:
    - not:
        - type: flow-logs
          enabled: true

18.- Confirm the VPC default security group does not allow inbound and outbound traffic 🟩

The rules for the default security group allow all outbound and inbound traffic from network interfaces (and their associated instances) that are assigned to the same security group.

We do not recommend using the default security group. Because the default security group cannot be deleted, you should change the default security group rules setting to restrict inbound and outbound traffic. This prevents unintended traffic if the default security group is accidentally configured for resources, such as EC2 instances.

Get the description of the default security group within the selected region:

aws ec2 describe-security-groups
--region REGION
--filters Name=group-name,Values='default'
--output table
--query 'SecurityGroups[*].IpPermissions[*].IpRanges'

If this command does not return any output, then the default security group does not allow public inbound traffic. Otherwise, it should return the inbound traffic source IPs defined, as in the following example:

------------------------
|DescribeSecurityGroups|
+----------------------+
|        CidrIp        |
+----------------------+
|  0.0.0.0/0           |
|  ::/0                |
|  1.2.3.4/32          |
|  1.2.3.5/32          |
+----------------------+

If the IPs returned are 0.0.0.0/0 or ::/0, then the selected default security group is allowing public inbound traffic. We’ve explained previously what the real threats are when securing SSH on EC2.

To remediate this issue, create new security groups and assign those security groups to your resources. To prevent the default security groups from being used, remove their inbound and outbound rules.

19.- Enable EBS default encryption 🟥🟥🟥

When encryption is enabled for your account, Amazon EBS volumes and snapshot copies are encrypted at rest. This adds an additional layer of protection for your data. For more information, see Encryption by default in the Amazon EC2 User Guide for Linux Instances.

Note that following instance types do not support encryption: R1, C1, and M1.

Run the get-ebs-encryption-by-default command to know whether EBS encryption by default is enabled for your AWS cloud account in the selected region:

aws ec2 get-ebs-encryption-by-default
--region REGION
--query 'EbsEncryptionByDefault'

If the command returns false, the encryption of data at rest by default for new EBS volumes is not enabled in the selected AWS region. Fix it with the following command:

aws ec2 enable-ebs-encryption-by-default
--region REGION

AWS Database Migration Service (DMS)

AWS Database Migration Service (AWS DMS) is a cloud service that makes it easy to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud or between combinations of cloud and on-premises setups.

20.- Verify AWS Database Migration Service replication instances are not public 🟥🟥🟥

Ensure that your Amazon Database Migration Service (DMS) is not publicly accessible from the Internet in order to avoid exposing private data and minimize security risks. A DMS replication instance should have a private IP address and the Publicly Accessible feature disabled when both the source and the target databases are in the same network that is connected to the instance’s VPC through a VPN, VPC peering connection, or using an AWS Direct Connect dedicated connection.

Sign in to AWS Management Console at https://console.aws.amazon.com/dms/.
In the left navigation panel, choose Replication instances.
Select the DMS replication instance that you want to examine to open the panel with the resource configuration details.
Select the Overview tab from the dashboard bottom panel and check the Publicly accessible configuration attribute value. If the attribute value is set to Yes, the selected Amazon DMS replication instance is accessible outside the Virtual Private Cloud (VPC) and can be exposed to security risks. To fix it, do the following:
1. Click the Create replication instance button from the dashboard top menu to initiate the launch process.
2. On Create replication instance page, perform the following:
  1. Uncheck Publicly accessible checkbox to disable the public access to the new replication instance. If this setting is disabled, Amazon DMS will not assign a public IP address to the instance at creation and you will not be able to connect to the source/target databases outside the VPC.
  2. Provide a unique name for the new replication instance within the Name box, then configure the rest of the instance settings using the configuration information copied at step No. 5.
  3. Click Create replication instance to launch your new Amazon DMS instance.
3. Update your database migration plan by developing a new migration task to include the newly created AWS DMS replication instance.
4. To stop adding charges for the old replication instance:
  1. Select the old DMS instance, then click the Delete button from the dashboard top menu.
  2. Within the Delete replication instance dialog box, review the instance details then click Delete to terminate the selected DMS resource.
Repeat step Nos. 3 and 4 for each AWS DMS replication instance provisioned in the selected region.
Change the region from the console navigation bar and repeat the process for all the other regions.

Learn more about security best practices for AWS Database Migration Service.

Amazon Elastic Block Store (EBS)

Amazon Elastic Block Store (Amazon EBS) provides block level storage volumes for use with EC2 instances. EBS volumes behave like raw, unformatted block devices. You can mount these volumes as devices on your instances. EBS volumes that are attached to an instance are exposed as storage volumes that persist independently from the life of the instance. You can create a file system on top of these volumes, or use them in any way you would use a block device (such as a hard drive).

You can dynamically change the configuration of a volume attached to an instance.

21.- Ensure Amazon EBS snapshots are not public, or to be restored by anyone 🟥🟥🟥

EBS snapshots are used to back up the data on your EBS volumes to Amazon S3 at a specific point in time. You can use the snapshots to restore previous states of EBS volumes. It is rarely acceptable to share a snapshot with the public. Typically, the decision to share a snapshot publicly was made in error or without a complete understanding of the implications. This check helps ensure that all such sharing was fully planned and intentional.

Get the list of all EBS volume snapshots:

aws ec2 describe-snapshots
--region REGION
--owner-ids ACCOUNT_ID
--filters Name=status,Values=completed
--output table
--query 'Snapshots[*].SnapshotId'

For each snapshot, check its createVolumePermission attribute:

aws ec2 describe-snapshot-attribute
--region REGION
--snapshot-id SNAPSHOT_ID
--attribute createVolumePermission
--query 'CreateVolumePermissions[]'

The output from the previous command returns information about the permissions for creating EBS volumes from the selected snapshot:

{
    "Group": "all"
}

If the command output is "Group": "all", the snapshot is accessible to all AWS accounts and users. If this is the case, take your time to run this command to fix it:

aws ec2 modify-snapshot-attribute
--region REGION
--snapshot-id SNAPSHOT_ID
--attribute createVolumePermission
--operation-type remove
--group-names all

Amazon OpenSearch Service

Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. Amazon OpenSearch Service is the successor to Amazon Elasticsearch Service and supports OpenSearch and legacy Elasticsearch OSS (up to 7.10, the final open source version of the software). When you create a cluster, you have the option of which search engine to use.

22.- Ensure Elasticsearch domains have encryption at rest enabled 🟥🟥🟥

For an added layer of security for your sensitive data in OpenSearch, you should configure your OpenSearch to be encrypted at rest. Elasticsearch domains offer encryption of data at rest. The feature uses AWS KMS to store and manage your encryption keys. To perform the encryption, it uses the Advanced Encryption Standard algorithm with 256-bit keys (AES-256).

List all Amazon OpenSearch domains currently available:

aws es list-domain-names --region REGION

Now determine if data-at-rest encryption feature is enabled with:

aws es describe-elasticsearch-domain
--region REGION
--domain-name DOMAIN_NAME
--query 'DomainStatus.EncryptionAtRestOptions'

If the Enabled flag is false, the data-at-rest encryption is not enabled for the selected Amazon ElasticSearch domain. Fix it with:

aws es create-elasticsearch-domain
--region REGION
--domain-name DOMAIN_NAME
--elasticsearch-version 5.5
--elasticsearch-cluster-config InstanceType=m4.large.elasticsearch,InstanceCount=2
--ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=200
--access-policies file://source-domain-access-policy.json
--vpc-options SubnetIds=SUBNET_ID,SecurityGroupIds=SECURITY_GROUP_ID
--encryption-at-rest-options Enabled=true,KmsKeyId=KMS_KEY_ID

Once the new cluster is provisioned, upload the existing data (exported from the original cluster) to the newly created cluster.

After all the data is uploaded, it is safe to remove the unencrypted OpenSearch domain to stop incurring charges for the resource:

aws es delete-elasticsearch-domain
--region REGION
--domain-name DOMAIN_NAME

Amazon SageMaker

Amazon SageMaker is a fully-managed machine learning service. With Amazon SageMaker, data scientists and developers can quickly build and train machine learning models, and then deploy them into a production-ready hosted environment.

23.- Verify SageMaker notebook instances do not have direct internet access 🟨🟨

If you configure your SageMaker instance without a VPC, then, by default, direct internet access is enabled on your instance. You should configure your instance with a VPC and change the default setting to Disable — Access the internet through a VPC.

To train or host models from a notebook, you need internet access. To enable internet access, make sure that your VPC has a NAT gateway and your security group allows outbound connections. To learn more about how to connect a notebook instance to resources in a VPC, see “Connect a notebook instance to resources in a VPC” in the Amazon SageMaker Developer Guide.

You should also ensure that access to your SageMaker configuration is limited to only authorized users. Restrict users’ IAM permissions to modify SageMaker settings and resources.

Sign in to the AWS Management Console at https://console.aws.amazon.com/sagemaker/.
In the navigation panel, under Notebook, choose Notebook instances.
Select the SageMaker notebook instance that you want to examine and click on the instance name (link).
On the selected instance configuration page, within the Network section, check for any VPC subnet IDs and security group IDs. If these network configuration details are not available, instead the following status is displayed: “No custom VPC settings applied.” The notebook instance is not running inside a VPC network, therefore you can follow the steps described in this conformity rule to deploy the instance within a VPC. Otherwise, if the notebook instance is running inside a VPC, check the Direct internet access configuration attribute value. If the attribute value is set to Enabled, the selected Amazon SageMaker notebook instance is publicly accessible.

If the notebook has direct internet access enabled, fix it by recreating it with this CLI command:

aws sagemaker create-notebook-instance
--region REGION
--notebook-instance-name NOTEBOOK_INSTANCE_NAME
--instance-type INSTANCE_TYPE
--role-arn ROLE_ARN
--kms-key-id KMS_KEY_ID
--subnet-id SUBNET_ID
--security-group-ids SECURITY_GROUP_ID
--direct-internet-access Disabled

AWS Lambda

With AWS Lambda, you can run code without provisioning or managing servers. You pay only for the compute time that you consume — there’s no charge when your code isn’t running. You can run code for virtually any type of application or backend service — all with zero administration.

Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

It is important to mention the problems that could occur if we do not secure or audit the code we execute in our lambda functions, as you could be the initial access for attackers.

24.- Use supported runtimes for Lambda functions 🟨🟨

This AWS security best practice recommends checking that the Lambda function settings for runtimes match the expected values set for the supported runtimes for each language. This control checks function settings for the following runtimes: nodejs16.x, nodejs14.x, nodejs12.x, python3.9, python3.8, python3.7, ruby2.7, java11, java8, java8.al2, go1.x, dotnetcore3.1, and dotnet6.

The AWS Config rule ignores functions that have a package type of image.

Lambda runtimes are built around a combination of operating system, programming language, and software libraries that are subject to maintenance and security updates. When a runtime component is no longer supported for security updates, Lambda deprecates the runtime. Even though you cannot create functions that use the deprecated runtime, the function is still available to process invocation events. Make sure that your Lambda functions are current and do not use out-of-date runtime environments.

Get the names of all Amazon Lambda functions available in the selected AWS cloud region:

aws lambda list-functions
  --region REGION
  --output table
  --query 'Functions[*].FunctionName'

Now examine the runtime information available for each functions:

aws lambda get-function-configuration
  --region REGION
  --function-name FUNCTION_NAME
  --query 'Runtime'

Compare the value returned with the updated list of Amazon Lambda runtimes supported by AWS, as well as the end of support plan listed in the AWS documentation.

If the runtime is unsupported, fix it to use the latest runtime version. For example:

aws lambda update-function-configuration
  --region REGION
  --function-name FUNCTION_NAME
  --runtime "nodejs16.x"

AWS Key Management Service (AWS KMS)

AWS Key Management Service (AWS KMS) is an encryption and key management service scaled for the cloud. AWS KMS keys and functionality are used by other AWS services, and you can use them to protect data in your own applications that use AWS.

25.- Do not unintentionally delete AWS KMS keys 🟨🟨

KMS keys cannot be recovered once deleted. Data encrypted under a KMS key is also permanently unrecoverable if the KMS key is deleted. If meaningful data has been encrypted under a KMS key scheduled for deletion, consider decrypting the data or re-encrypting the data under a new KMS key unless you are intentionally performing a cryptographic erasure.

When a KMS key is scheduled for deletion, a mandatory waiting period is enforced to allow time to reverse the deletion if it was scheduled in error. The default waiting period is 30 days, but it can be reduced to as short as seven days when the KMS key is scheduled for deletion. During the waiting period, the scheduled deletion can be canceled and the KMS key will not be deleted.

List all Customer Master keys available in the selected AWS region:

aws kms list-keys --region REGION

Run the describe-key command for each CMK to identify any keys scheduled for deletion:

aws kms describe-key --key-id KEY_ID

The output for this command shows the selected key metadata. If the KeyState value is set to PendingDeletion, the key is scheduled for deletion. But if this is not what you actually want (the most common case), unschedule the deletion with:

aws kms cancel-key-deletion --key-id KEY_ID

Amazon GuardDuty

Amazon GuardDuty is a continuous security monitoring service. Amazon GuardDuty can help to identify unexpected and potentially unauthorized or malicious activity in your AWS environment.

26.- Enable GuardDuty 🟨🟨

It is highly recommended that you enable GuardDuty in all supported AWS Regions. Doing so allows GuardDuty to generate findings about unauthorized or unusual activity, even in Regions that you do not actively use. This also allows GuardDuty to monitor CloudTrail events for global AWS services, such as IAM.

List the IDs of all the existing Amazon GuardDuty detectors. A detector is an object that represents the AWS GuardDuty service. A detector must be created in order for GuardDuty to become operational:

aws guardduty list-detectors
--region REGION
--query 'DetectorIds'

If the list-detectors command output returns an empty array, then there are no GuardDuty detectors available. In this instance, the Amazon GuardDuty service is not enabled within your AWS account. If this is the case, create a detector with the following command:

aws guardduty create-detector
--region REGION
--enable

Once the detector is enabled, it will start to pull and analyze independent streams of data from AWS CloudTrail, VPC flow logs, and DNS logs in order to generate findings.

AWS compliance standards & benchmarks

Setting up and maintaining your AWS infrastructure to keep it secure is a never-ending effort that will require a lot of time.

For this, you will be better off following the compliance standard(s) relevant to your industry, since they provide all the requirements needed to effectively secure your cloud environment.

Because of the ongoing nature of securing your environment and complying with a security standard, you might also want to recurrently run policies, such as CIS Amazon Web Services Foundations Benchmark, which will audit your system and report any non-conformity it finds.

Conclusion

Going all cloud opens a new world of possibilities, but it also opens a wide door to attacking vectors. Each new AWS service you leverage has its own set of potential dangers you need to be aware of and well prepared for.

Luckily, cloud native security tools like Falco and Cloud Custodian can guide you through these best practices, and help you meet your compliance requirements.

Blackhat 2022 recap – Trends and highlights

Miguel — Tue, 16 Aug 2022 12:59:41 +0000

Blackhat 2022, on its 25th anniversary, took place this week in Las Vegas. It’s the most important event for the infosec community and the best place for security vendors to showcase all their innovations and products in this ever-growing ecosystem. This year, attendees come from 111 different countries.

In 2020, Black Hat added the word Cloud to the existing track about Platform Security. Research on Kubernetes, containers, and other cloud technologies had been presented at Black Hat before 2020, but adding Cloud to the track title really highlighted the importance of securing the cloud that many corporations and individuals rely upon every day.

In summary, we have cloud at the core of most of the talks, eBPF is growing in visibility, and attacking and defending the supply chain as a priority. As a threat, we could say ransomware in the cloud and global conflict management, and finally the importance of early detection of burnout.

In this Blackhat 2022 recap, we’ll share our insights about the talks that we believe to be the most interesting.

Blackhat 2022 – KEYNOTES

In this edition, one of the main topics in the keynotes was the control of information, the problem with how fake news is impacting our lives, and the difficulties of checking reliable information. The solution, in some cases, is to intensify controls on information sharing, but this could be used for censorship. Now it’s not just governments and companies that are involved. New players in the field, individuals, are part of it, and like spam detectors, we need something ethical to protect us.

The invasion of Ukraine is another topic mentioned in the first few minutes of Blackhat 2022. Some companies started to participate, and banned relations with Russian projects or affected domain renewal in Russia. So, the end of these conflicts is all but certain, and instead it seems that they are progressively more chaotic with increasing impacts on society.

Chris Krebs, Director of CISA, explained in one of the Blackhat 2022 keynotes the main four reasons for why it is so bad right now and left us with an important lesson: “Life is too short to work for a**holes.”

Technology: More software means more complexity and, by default, more insecure code integration to maintain and update. In addition, Krebs emphasized how the lack of vision within the cloud causes complex risk management.
Bad actors: Attackers know all this and know the profit is in the cloud. As we mentioned in the anatomy of cloud attacks, cloud ransomware is the future threat and is going to have a big impact. Attackers are betting on targeting the supply chain to gain access.
Government: Regulation, economics, and innovation are growing, but it is very difficult to work with governments efficiently. Compliance must change to get more results than a checklist. We must watch out for future global incidents to be prepared.
People: How do we balance all this information? In most cases, people are the end users of all this technology and the victims of breaches and incidents.

Kim Zetter, investigative journalist, gave a good overview of the evolution of threats and how this impacts companies. The media has had to adapt when it comes to explaining new vulnerabilities or threats to the audience. She mentioned the Pre-Stuxnet and Post-Stuxnet eras, where everything has changed but the same actors remain. She started with ransomware, evolving into more sophisticated malware that modified code to eventually directly affect the supply chain with SolarWinds.

In a real case, she explained that a company with a perfect backup system and everything prepared to be restored (in case it was affected by ransomware) paid the ransom when it happened. Why? It never tested the recovery process. Therefore, we should anticipate and perform simulations just like we do fire drills.

Finally, we all know the good security practices, such as isolating networks, reinforcing perimeters, mfa, IR plan and backups, that protect us from most attacks. But, critical infrastructure is still not fully implemented, remaining exposed and without authentication on the internet.

Blackhat 2022 – Top Briefings

During two days of Blackhat 2022 USA trending sessions, we were able to enjoy several high-level talks on cybersecurity. These are, in our opinion, the most remarkable ones.

IAM The One Who Knocks

One of the most important parts of securing your cloud account is how to manage identity access. In this talk, Igal and Noam provide a good explanation of how each public cloud (AWS, GCP, and Azure) implements access and authorization. Every cloud provider works differently but there isn’t one that is clearly better than the others; the major difference is the scope of the permission. AWS scope is part of the policy itself, and they recommend AWS SSO.

Another important topic was the non-human identities. Each provider calls them something different, but the attackers don’t care. They only like to exploit the Cloud IAM weaknesses and the speakers explain how with different examples.

Here are the takeaways.

Default is the best friend of attackers: Maintaining a good CSPM is crucial to stop the majority of attacker’s attempts to control all your accounts. Avoid temporary fixes that become permanent.
Monitor and protect credentials: Not only is the creation part relevant, but the modification of a compromised account is a huge threat.
Log everything but know the limits: the golden rule of security is to log everything, but in some cases we exceed the limits. Attackers use this to hide their actions and go unnoticed. At this point, we want to highlight another option. Try to detect at runtime or at the time when these logs occur to avoid the large amount of logs (only one window is sufficient if the initial compromise attack is detected). That is what Falco open source tries to do.

Finally, the speakers shared different open source tools that will help you manage excessive permissions. Two types of tools, the constructive (adding only the requirements, the principle of least privilege) and the reductive (default permission and remove from it).

Constructive	Reductive
policy-sentry	Cloudtracker
iamlive	Repokid
access-undenied-aws	IamSpy
	PMapper
	Cloudsplaining

Trying to Be Everything to Everyone: Let’s Talk About Burnout

Burnout is no joke. You have to take care of yourself and detect burnout before it is too late. This talk exposed the problems within the security industry, including the high levels of mental workload, the rapid response that security engineers are forced to have when a shortage occurs, and when they must anticipate cyber attacks that lead to maintaining a high level of stress.

Of course, it’s very difficult to quantify security, so it’s hard to know if we’re investing our time and effort (and money!) well. For this reason, we can often feel that we are not doing our job well no matter how hard we try.

The speaker suggests good health, physical fitness, and relaxation techniques. The most common solutions for burnout are to work in short time slots, balance it with your personal life or, if you can, stop working and be free. Joking aside, ask for help at the slightest risk of burnout.

RCE-as-a-Service: Lessons Learned from 5 Years of Real-World CI/CD Pipeline Compromise

We have to go back to basics; this is one of the points to keep in mind. The speakers explained different use cases and examples where the CI/CD pipeline was compromised based on issues that are not new. Examples include exposed credentials encoded in YAML or other configuration files, failure to isolate networks, or not following the principle of least privilege. One of the demonstrations focused on cloud-to-premises hopping, an increasingly common threat.

To solve this problem, one of the things mentioned is the signing process, verify everything we are running. We explain in detail these procedures in Kubernetes by using cosign and connaisseur.

Trace Me if You Can: Bypassing Linux Syscall Tracing

System call traces are used to detect different behaviors within linux systems. There are several tools to obtain this data and process it to generate alerts when something strange happens. Like the Kubecon talk (Bypassing Falco), this talk focuses on the Falco tool and the TOCTOU problem solved in the current version.

TOCTOU, or time-to-check to time-to-user, is a software bug caused by a race condition. In this talk, the speaker simulates how the attacker could modify the trace of a syscall in several scenarios. Here, we summarize two of them:

The first uses delay injection in the communication between the client (or compromised device) and the server (in this scenario, the C2C). The technique is based on delaying the response in the handshake when the communication is initiated. The communication, when the exploit replaces the original IP with a fake one, is hidden from post-forensic analysis. If we check the network traces in Wireshark, we can discover the real IP of the C2C in the first packets.
The second scenario is the same idea but on the routes we like to modify. The attack is performed to store information inside GKE, avoiding detection by changing the path of the file we are writing.

The demo is performed exploiting using Phantom-attack, presented at DefCon 29. The mitigation was using the new version of Falco (>0.32), and it is also recommended to use other methods to block the system call such as seccomp. This would affect its performance.

Kubernetes Privilege Escalation: Container Escape == Cluster Admin?

This talk explains the same concept presented at the KubeCon EU with the talk Trampoline Pods.

The scenario presented is as follows: inside our Kubernetes cluster we have containers running, but one of these containers is vulnerable and the attacker gains access to execute commands and control the entire node using container escape techniques.

What is the next step to control the entire cluster? Kubelet’s credentials are not enough, and the node permissions are different to be admin. The goal is to get control of a trampoline pod in DaemonSets to perform lateral moves between nodes and obtain control of the whole cluster. There is no clear list of what permissions are required to carry out this type of attack, but depending on which ones are required, one or the other can be performed.

AuthN/AuthZ: Impersonate to escalate roles.
Acquire Tokens: Create new tokens for privilege escalation and, with the service account, enable secret list and use it to authenticate to the api-server.
RCE: You do not need to escalate privileges to run your code, it depends on your configuration. It controls Kubelet.
Steal Pods: Move the pod from one node to another with a powerful services account. Update nodes or delete other pods permission required.

This Trampoline pod affects all major cloud providers with the default behavior and, for this, escaping the container means managing the entire cluster. They shared the tool rbac-policy that was presented in KubeCon EU.

Cautious: A New Exploitation Method! No Pipe but as Nasty as Dirty Pipe

The new exploitation method presented by Zhenpeng Lin is called DirtyCred and is based on swapping Linux Kernel credentials. It’s simple, effective and generic. This allows it to perform container escape and still be an actual threat.

Exploitation is performed on a system vulnerable to CVE-2021-4154 or CVE-2022-2588, and the speaker explained two path attacks.

Attacking task credentials (struct cred): Our un-privileged credentials inside the kernel heap are modified, free, and privileged credentials are put in the same place to impersonate them.

Attacking open file credentials (struct file): Free the file after checks but before writing to memory. Allocate a read-only file object in the freed memory slot.

To do this deterministic and not wait for privileged users to allocate task credentials, the attacker could trigger privileged processes in user space (executables with root SUIDs or daemons running as root). Finally, the attacker needs to stabilize the file exploit by extending userfaultfd or FUSE (pause kernel execution).

It’s a very interesting exploitation method that requires further research.

DNSSEC Downgrade Attacks

DNSSEC is the solution to DNS, based on cryptography signatures and verification of the resolver with a chain of trust. DNSSEC protects us from DNS Poisoning when we have a compromised application. DNSSEC provides data origin authenticity and integrity, but not confidentiality.

The attack is based on making the resolver use the weakest security path and attacking that weakest link in the chain of trust. The final model is that the attacker, without knowing the cryptographic secrets, is able to activate the resolver, intercept it, and modify the records and signatures. The result? Successful impersonation of a real resolver.

Countermeasures against this are to require the stronger DS and to drop SHA-1.

If you are interested in knowing more about DNS and how to configure it in a secure way in the cloud, read the article How to protect DNS in the Cloud.

Scaling the Security Researcher to Eliminate OSS Vulnerabilities Once and For All

The speakers started with a simple concept. When we detect a vulnerability in open source code, how do we fix it? Well, fork the code and PR with changes inside that resolve the vulnerabilities. But what if this insecure code is in thousands of repositories?

The first vulnerability that was fixed is the download or update of dependencies via HTTP, which are vulnerable to MiTM and not recommended at all. But is it a real problem? Well, Sonatype maven core downloads still use HTTP (25%).

To solve this problem, the speaker creates a python bot with CodeQL and scans 100k with a simple regex.

With this, the bot created over 1,400 pull requests with an acceptance rate of 40%. Pretty impressive. But this is just the beginning. How can it do the same with all the security vulnerabilities within the repository code on a massive scale?

Well, openRewrite has been introduced, an automated software refactoring to keep up to date with API changes, fix vulnerabilities, and improve code quality and the Moderne platform to manage the entire process. The talk explained three vulnerabilities (temporary directory hijacking, Partial path traversal, and Zip Slip) and focused on java language.

What’s next?

This has been the most relevant at Blackhat 2022 USA. The main topics are still Kubernetes security, cloud security, and supply chain attacks, but we have a lot of global incident presence.

In a few months, we will have the next Blackhat in Europe. We hope to be here with more great talks and demos.

Understanding CVSS score: Are vulnerability scores misleading you?

Miguel — Wed, 20 Jul 2022 14:16:13 +0000

Vulnerabilities are everywhere. Vetting, mitigating, and remediating them at scale is exhausting for security practitioners. Let’s keep in mind that no organization has the capacity to find and fix all vulnerabilities. The key is to understand what a vulnerability is, interpret the meanings of the CVSS score, and prioritize and effectively use resources within constrained time limits or delivery windows.

Since 2016, new vulnerabilities reported each year have nearly tripled. As of April 2022, predictions about the number of new vulnerabilities continue to come true.

The trend continues to increase. Basically, more code translates to more vulnerabilities. And code now takes many forms beyond just applications or software. Code exists within embedded systems and IoT devices, resulting in hardware-born vulnerabilities, and code is also used to define and operate infrastructure as part of DevOps practices. It is now commonplace among security engineers or analysts to be accustomed to the Internet being broken every week. The shortage of developers is not helping either.

Vulnerabilities by the last years

All critical vulnerabilities are not the equivalent of log4j or spring4shell in terms of widespread adoption of a software package, exploitability, or impact.

The ideal state for any cybersecurity program is to be able to quickly identify vulnerabilities that are truly impacting the organization and are actionable. Burning out IT teams and security teams by chasing all vulnerabilities is untenable.

In this article, we want to explain:

What is a vulnerability?
What is the meaning of the CVSS score?
What variables impact CVSS scores?
Is CVSS the best way to prioritize within vulnerability management?
Are there alternatives to CVSS for risk-based prioritization?

What is the meaning of vulnerability?

MITRE defines a vulnerability as:

“A weakness in the computational logic (e.g., code) found in software and hardware components that, when exploited, results in a negative impact to confidentiality, integrity, or availability.”

For this reason, it is possible to have a critical vulnerability in a code that does not affect you at all, for example, because this code runs on an IoT device that relies on other security controls that effectively mitigate exploitability of a latent vulnerability in the embedded code.

On the other hand, you may have a vulnerability of low severity that negatively impacts availability of your application, such as a denial of service. You’d likely prioritize fixing it as soon as possible, because fixing the issue directly impacts your business model.

As we mentioned before, the main problem is that we are continuously fed with new vulnerabilities while still wrestling with old vulnerabilities, and there is no easy way to manage them all. We have to be quick in detection and resolution processes when something really critical is discovered and put a majority of our efforts there without forgetting the rest of the vulnerability ecosystem. It sounds simple in theory and underpins all modern security programs, but vulnerability prioritization in practice is now one of the biggest gaps in security.

To go deeper into vulnerability management, we will explain what the process is when a vulnerability is released.

The lifecycle of a vulnerability

The origin of a vulnerability is not defined by anyone.

Sometimes, it is large research by a company regularly testing their own code that spends great efforts to show the problem with an application or the abuse of a dependency. At other times, a vulnerability might be discovered as a result of an independent security researcher probing a system in their free time and reporting the findings as part of responsible disclosure, or by creating a proof of concept (PoC) to exploit a system and publishing the details on Twitter.

These, before being officially published, are common examples of 0-days. It is an overloaded buzzword because if the vulnerability is already known, it is not a 0-day. This is why these types of vulnerabilities are gold in the black markets. If an attacker has intimate knowledge of a previously unidentified vuln, they can then exploit it readily since there’s likely a lack of detection and protection mechanisms in place for most organizations, at least in the initial access or exploitation phase.

A good practice that is done by researchers is to give developers some time to start working on the patch to fix the vulnerability before registering. Otherwise, days could go by without a fix.

What is the best thing to do then? Improve the process to be ready as soon as possible when a 0-day is disclosed and detect it from that moment on, providing the appropriate mitigations and in many cases, verifying that this vulnerability has not been used in the past (where it really was 0-day).

When the vulnerability is registered, we have an ID that identifies it. This will help us to identify the vulnerability and check if we are being impacted or not. But where is it registered?

One of the most common sites, but not the only one, is Common Vulnerabilities and Exposures (CVE). MITRE Corporation is the organization that identifies, defines, and catalogs publicly disclosed cybersecurity vulnerabilities and shares that information of CVE-IDs publicly. Vulnerability information is also shared with the NIST organization, where additional information may be added on to provide further details or security guidance. That information lives within NIST’s National Vulnerability Database (NVD) and is organized by CVE-IDs.

Other states have their own system to catalog and store their vulnerabilities, such as the Chinese National Vulnerability Database (CNNVD) or Japan Vulnerabilities Notes (JVN). But in this article, we focus on the NVD.

Once we have confirmation that the vulnerability is real, exploitable, and has an ID, the next process is to assess the severity.

How the score of a vulnerability is calculated

The Common Vulnerability Scoring System (CVSS) provides a way to capture the key characteristics of a vulnerability and produce a numerical score that reflects its severity. Many security teams and SOCs use the CVSS to prioritize vulnerability management activities, such as incident response processes, defect tracking and resolution, or implementation of a mitigating control.

The metrics used in CVSS v3.1, the latest version, assess the different elements that depend on the exploitation process and the impact, resulting in the final severity score. The first thing we can find in the documentation is that CVSS measures severity, not risk.

CVSS, as scored, is an “objective” score when you set some attributes of the vulnerability without context, and a formula produces a score that also maps to a “Severity.” Below, we can see a real example of the CVSS of Spring4Shell vulnerability, which scores the severity in 9.8 CRITICAL.

The base score is calculated with eight variables:

Attack Vector (AV): There are four options that represent the access method to exploit the vulnerability. The network is the most valued because it allows the remote and that implies that the attacker can exploit it from any location. The rest go from highest to lowest in restriction. In the case of Local, the vulnerable component is not bound to the network stack and the attacker’s path is via hardware access, remote, or hijacking the identity of an authorized user through social engineering.
Attack complexity (AC): There are two options, low or high. In version 3.1, it was updated and depends on the system requirements to be vulnerable, where there can be a debate if a configuration is taken as likely or unlikely. In the case of high complexity, it is defined as a successful attack which depends on conditions beyond the attacker’s control, but requires the attacker to expend a measurable amount of effort in preparation or execution against the vulnerable component. For example, the attacker needs to use brute force attacks to win a race condition, and not a silver bullet like log4j with a single command.
Privileges Required (PR): There are three options. None is when the vulnerability can be exploited without authentication. This also makes it difficult for the attribution or the path followed by the attacker after exploitation. If this metric is high, the attacker requires admin privileges or something similar to affect allowing access to component-wide settings, for example.
User interaction (UI): If it requires interaction, it is a lower score. This is common in mobile applications where the user needs to interact with the threat (malware) in order to breach their device. Another example, similar to a phishing attack, in itself is not a risk, but the attacker uses social engineering to get the victim to click on the link and be pwned.
Scope (S): The score will depend on whether the vulnerability affects only a specific component or affects the entire application.
CIA (Confidentiality, Integrity and Availability): The CIA triad, a respected security model that forms the basis for the development of security systems and policies. Here is the real impact of the vulnerability. The other part is the process of exploitation. With an RCE, the attacker has full control of the victim machine, and if the privileges are sufficient, it affects all three parties with high severity.

The final format of CVE-2022-22965 is a vector with this information: CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H

This first part corresponds to the base score, an objective value that should remain stable over time and consistent across organizations. As a supplement, there are two more metrics, temporal and environment; these values introduce more scoring complexity though and may not be something your organization chooses to pay attention to in the early phases of vulnerability management.

The Temporal metrics measure the current state of exploit techniques or proof of concept code availability, the existence of any patches or workarounds, and the confidence in the description of a vulnerability. It is something that will change along the lifecycle of the vulnerability because there’s a huge difference between having or not having the remediation ready. Environment metrics enable the practitioners to customize the CVSS score depending on the importance or business criticality of the affected IT asset to the impacted organization.

Vendors, such as RedHat or Debian as the base distributor provider, will also evaluate the severity of the vulnerability in a specific context (i.e., the package inside the distribution). Customers may trust the score of the vendor more than the generic scores assigned by MITRE or NIST, as it is usually more accurate.

As we can see, this score is not impacted by the remediation part or fix process. If this vulnerability needs a great effort to be solved, it does not impact the final score. In addition, two vulnerabilities with the same score could have a very different impact or likelihood because they occur in the economic sector or business vertical.

From the CVSS score calculation, several derivations appear that can be of help when evaluating the safety of a system. Some of them are:

Common Misuse Scoring System (CMSS): It is a set of measures of the severity of software featuring misuse vulnerabilities. Misuse vulnerabilities allow attackers to use the functionality that was intended to be beneficial for malicious purposes. This score could assist companies to provide data to be used in quantitative assessments of the overall security posture of a system.
Common Configuration Scoring System (CCSS): CCSS is based on CVSS and CMSS. The most notable difference is the type of exploitation: active or passive. Active exploitation refers to an attacker performing actions to take advantage of a misconfiguration, while passive exploitation refers to a misconfiguration that prevents authorized actions from occurring, such as a configuration setting that prevents audit log records from being generated for security events.
Common Weakness Scoring System (CWSS): Conceptually, CVSS and CWSS are quite similar. CWSS can be applied in the early process of releasing a new vulnerability. In addition, it can serve to supplement the lack of some information in the vulnerability report. Because the conservative approach is to inflate the scores, a deep understanding of the affected technology provides some of this unavailable information. When a new vulnerability is reported, it is possible to report it together with the exploited week, CWE-ID / CWSS in the same way as with CVSS.

It is obvious that we must feed our systems with more information to correlate with the CVSS and improve our vulnerability management. Remember that risk-based prioritization is the goal of all modern cybersecurity programs.

Using CVSS Score Alternatives to Prioritize Security Risk

If we dig deeper into the meaning of vulnerability severity, we may be more interested in other characteristics when calculating the CVSS score.

Obviously, depending on the use case or business sector, it is possible to find alternatives to the CVSS to help you prioritize the management of your vulnerabilities. It’s not always possible to patch quickly enough, especially in cases of third-party code or partner integrations. In this case, the shift-left approach is not enough and we recommend the use of runtime security as another layer of security that enables early detection and identification of affected software, expediting the implementation of a mitigating security control.

Exploit Prediction Scoring System

What is the actual probability of a vulnerability being exploited by an attacker? That probability is explained by the Exploit Prediction Scoring System (EPSS). The EPSS model produces a probability score that, the higher the score, the greater the likelihood that a vulnerability will be exploited.

The score is maintained by the same organization as the CVSS, MITRE, which guarantees its consistency with the above-mentioned vulnerability taxonomies and classification systems. If we look at the highest rated vulnerabilities of the last 30 days, we better understand the potential real impact of vulnerabilities. An example can be seen here with CVE-2022-0441, which relates to the MasterStudy LMS WordPress plugin.

To calculate the percentage, the EPSS uses part of the CVSS score but also uses threat intelligence to see how easy it is to exploit the vulnerability. For example, an exploit might enable exploitation of other vulnerabilities to increase impact. As part of a complex attack chain, an attacker may achieve RCE by exploiting one vulnerability and can then exploit other vulnerabilities to elevate privileges, resulting in a much more significant impact. The score may also factor in availability of exploit tools or repositories, like Metasploit or Exploit-db, which don’t require knowledge about the exploitation steps.

Stakeholder-specific Vulnerability Categorization

The Stakeholder-specific Vulnerability Categorization (SSVC) is mostly a conceptual tool for vulnerability management. SSVC aims to avoid one-size-fits-all solutions in favor of a modular decision-making system with clearly defined and tested parts that vulnerability managers can select and use as appropriate to their context.

The goal of SSVC is to be risk-oriented, be more transparent in the calculation process, and be able to scale the quantification of vulnerability risk through automation.

Vendor’s scoring

Vulnerability Priority Rating (VPR) is maintained by Tenable and also uses the severity and the facility to be exploited, similar to EPSS.

The Vulnerability Priority Rating (VPR) is a dynamic companion to the data provided by the vulnerability’s CVSS score since Tenable updates the VPR to reflect the current threat landscape, such as the exploit code of a vulnerability becoming available or having escalated maturity. The VPR values range is from 0.1 to 10.0, with a higher value representing a higher likelihood of exploitation.

Other vendors such as Snyk created their own score (Snyk Priority Score) for prioritization by using CVSS and other factors mentioned above, such as exploit maturity, remediation process, or mentions in the community even they rank vulnerabilities as part of their own threat research that may not have CVE-IDs associated but provide value in prioritization.

Vertical-specific approaches

Relevant to the medical sector, Risk Scoring System for Medical Devices (RSS-MD) is being considered and at a more generic level. As expected, a vulnerability in this industry directly affects people’s health or safety, so it is necessary to have a scale of its own to manage this type of vulnerability and relative impacts differently.

Relevant to the manufacturing industry, Industrial Vulnerability Scoring System (IVSS) incorporates part of its calculation factors such as physical security, among others. This score is specifically designed for vulnerabilities in industrial control systems that affect critical infrastructure where damage can impact entire cities and lives of citizens.

Next step? Incorporating vulnerability feeds and driving remediation

The tendency, as seen above, is to calculate the best score for a vulnerability or the associated risk-based on correlating as much information as possible that can be accessed and processed in order to “vitaminize” the final result.

It is strange that one method contradicts another. Normally they will all have a similar view of the final severity, but these small differences are crucial in a huge scale of vulnerability management. The simplicity approach is worth stressing since some of these scoring mechanisms get incredibly complicated. Many orgs would benefit by keeping their risk-scoring simplified so they can focus their efforts on addressing security problems instead of burning cycles qualifying or quantifying risk.

It is also necessary to have complete visibility of your situation at all times to know if we are being impacted as soon as possible and effectively reduce the risk caused by them.

With all this information, we now need to implement our vulnerability management processes and supporting tooling in our organization.

These vulnerability scores can be viewed ad-hoc, but effective cybersecurity requires that you ingest vulnerability feeds into appropriate security tooling that serve the relevant stage of the system lifecycle.

The key is to be prepared for a new vulnerability and be flexible to close the gap between the vulnerability release and the detection process in your environment.

One of the most famous feeds is Vulndb, which uses National Vulnerability Database (NVD) as a trusted database of vulnerabilities and also owns registered vulnerabilities and collaborates with security companies to be as up-to-date as possible.

If you’re missing the explicit details of a vulnerability, you must still acknowledge there is a potential risk and then accept, avoid, or mitigate it. You need to have an alternative to being sensitive with it.

A significant hurdle to overcome with respect to remediation is to be able to quickly patch every single asset or dependency that is impacted or potentially exploitable, and these processes also need to be able to scale. It’s not trivial to patch old versions that could impact new features or poor performance; do it in a massive way or have other implications. The problem is exacerbated with transitive dependencies. That is, your code or system likely relies on many other codebases or systems, and dependency chains become quite nested in practice. Sometimes, it is even necessary to patch old versions that are still being distributed. That is what RedHat calls backporting.

Backporting

Backporting is essentially the management of updates through automation to minimize the associated risk. It is possible that a fix in a new version may adversely affect the previous version. You should be aware of when you want to upgrade as soon as possible.

When a vendor offers to backport security fixes, we ensure that fixes do not introduce unwanted side effects and apply to previously released versions.

Lessons learned about cvss score

Prioritization in a world with hundreds of vulnerabilities every day is a necessity. We continuously develop more software that will be targeted by attackers and add to the libraries, firmware, or common dependencies that are already used by applications and systems.

To help us, we need to ingest the vulnerability information that organizations like MITRE share, generate better indicators through the correlation of other sources of information, and maintain full visibility of our assets (and associated attack vectors) to be quick in detecting the impact. Without this, it is impossible to both efficiently plan the vulnerability mitigation process to reduce the noise and time in which we are vulnerable, and be effective in any cybersecurity program.

Manual processes can’t scale to infinity, and you’ll never have enough headcount for all your security needs. Security needs to be seamless and automated. Organizations must plan accordingly to keep ahead of the tide of critical vulnerabilities, like log4j and spring4shell.

Trends at Blackhat Asia 2022

Miguel — Wed, 25 May 2022 10:25:19 +0000

This week, BlackHat Asia 2022 took place in hybrid mode. It’s one of the most important events within the #infosec community, where security experts show how far they can go. In this edition, the trend of talks and tools focused on improving the security of Kubernetes, Cloud Security, and Supply Chain, either from the perspective of the blue team or the red team.

In this article, we’ll share our insights about a few talks and tools presented that we liked, and we’ll give you an idea of the future trends this year in cybersecurity.

Briefings

During two days of Blackhat Asia informative sessions, we were able to enjoy several high-level talks on cybersecurity. These are, in our opinion, the most remarkable ones.

Backdoor Investigation and Incident Response: From Zero to Profit
- Managing a security incident where a backdoor takes place is not trivial. This talk explains the Backdoor Incidence Response Matrix (BDIRM) framework based on a triangle (server, backdoor, and network) for the acquisition and analysis of data to understand the attacker’s access. This allows us to make a better attribution and generate the best indicators of compromise or detection techniques.
The Firmware Supply-Chain Security Is Broken: Can We Fix It?
- Dependencies are the headache of any security auditor or developer, and even more so when you don’t have full visibility. In some cases, firmware components are vulnerable and continue to be used because they are not exploitable on their own. That is why when another vulnerability appears in a different component, it makes a previous one possible, making it much more complex to see the risk of old vulnerabilities that remained latent and badly scored.
Using Zero to Attack Zero-Knowledge Proof (ZKP) PLONK
- This talk reviews an incredible but real case of theoretical vs practice. The speaker discusses a critical issue in a cutting-edge ZKP PLONK C++ implementation which allows an attacker to create a forged proof that all verifiers will accept.
Quantify Security Effectively – Moving the Security Needle From the Security Trenches to the Boardroom
- One of the keynotes. The speaker shared attracting ideas such as the definition of a shared responsibility model between developers and the cybersecurity team. Understanding who owns the vulnerability and who owns the mitigation is key to avoiding future incidents, loss of time, and money. It is necessary to escalate and prioritize, otherwise it is not achievable.
- Another impressive concept is to quantify success in cybersecurity. It is necessary to measure it and thus be able to check if the measures are being effective.
Like Lightning From the Cloud: Finding RCEs in an Embedded TLS Library and Toasting a Popular Cloud-connected UPS
- This talk explains the importance of handling errors in code. The presenters explained how the exploitation of this would allow an attacker to control switches and systems such as UPS (controls system power if the network goes down), and how to replicate the exploit in different vendors because they use the same implementation. During the demonstration, they provoked the burning of the device.
Dynamic Process Isolation
- Explanation of a remote Spectre attack using amplification techniques in combination with a remote timing server. The authors contribute with a process isolation mechanism that only isolates suspicious worker scripts following a detection mechanism. The Dynamic Process Isolation paper demonstrates a solution to detect all state-of-art of this kind of attack.

Arsenal

Several tools were presented at Blackhat Asia this time. Although not necessarily new, it is always interesting to see the latest features or discover unknown tools. Something to mention are the differences when changing the point of view. For instance, considering Kubernetes tools as intended for red teams against those of the supply chain where the focus is its usage by blue teams.

Kubesploit
- An open source penetration testing framework that can improve your cybersecurity posture scanning your cluster and also post-exploitation attacks. This tool is a must in your repository.
Kdigger
- This CLI tool is similar to the first one, but also recommended as it keeps adding improvements. To present the features, the demo shows a minik8s-ctf environment. It is really great to test and implement the new features.
ThunderCloud
- This tool is a compilation of other techniques focused on AWS. Two of them are especially interesting: creating a SSO phishing to steal the access token and the simple code to collect the ACCESS KEYs when the Cognito endpoint is known and misconfigured.
In Supply Chain Attacks, three tools were presented. Dependency Combobulator detects dependency confusion using heuristics; for example, if the repository is public or time since last change. Similar to packj but in this case, it implements metadata (if the repository activates 2FA) or typosquatting detection, finding packages with similar names to avoid errors. ChainAlert focuses on automation and detection of dependency commitment using the difference of tags between Github and NPM, but detection is very low.
Pwnppeteer is an offensive tool to manage the phishing attacks with lambda functions to automate the process
Telegrip assists in obtaining evidence from telegrams for android devices with an autopsy-like UI, a great forensic tool.

CSPM, CIEM, CWPP, and CNAPP: Guess who in cloud security landscape

Miguel — Thu, 05 May 2022 13:35:17 +0000

Your organization may be trying services in the cloud, running a few applications, or totally embracing this new era of cloud. Regardless if you are in the early stages or running all of your workloads in production, you have likely already noticed that cloud-native security is different from IT-managed data center security.

A recent Gartner survey found that 50 percent of participating organizations indicated that there is a lack of internal knowledge about cloud-native security.

Security teams are trying to figure out the right security solutions to use in the cloud, but the market is evolving too fast for them to keep up. So, what are those terms you keep hearing from vendors, cloud providers, and security training courses? What are the things you should focus on?

Gartner, Forrester, IDC, and 451 Group are some of the most well-known analyst firms that identify and describe emerging trends in the market and create definitions for new technologies. They have coined terms you know, like SIEM, CRM, and WAF. But we want to introduce you to new terms, such as CSPM, CWPP, and CIEM, among others.

One of the latest categories mentioned in Gartner’s Emerging Technologies: Future of Cloud-Native Security Operations is CNAPP. Where does this new term fit in? We can think of CNAPP as the convergence of CWPP, CSPM, and CIEM, plus some other goodies. I know that’s not a very helpful definition since you may not know what CWPP, CSPM, and CIEM mean yet, right?

Let’s find out step by step.

CWPP

It all started with DevOps teams moving their workloads to the cloud.

In order to secure the whole DevOps workflow, security leaders need to fulfill some specific use cases, and that’s what Cloud Workload Protection Platform (CWPP) tools focus on. They secure workloads, typically providing cloud-based security solutions that protect instances on AWS, Microsoft Azure, Google Cloud Platform (GCP), and other cloud vendors.

What are these use cases?

Runtime detection: Prevent and detect suspicious behavior at runtime in containers and microservices. Automate response for container threats.
System hardening: Detect anomalous activity inside of Linux hosts or VM-based workloads running on top of the host.
Vulnerability management: Detect OS and non-OS vulnerabilities from container images stored in CI/CD and registries before deploying to production.
Network security: Visualize network traffic inside containers and Kubernetes, and enforce Kubernetes-native network segmentation.
Compliance: Validate container compliance and ensure File Integrity Monitoring inside containers.
Incident Response: Conduct forensics and incident response for containers and Kubernetes even after the container is gone.

Those are the use cases that would fall under a Cloud Workload Protection Platform (CWPP) solution, and what a CWPP solution will handle, securing workloads across the application lifecycle.

CSPM

As the workloads moved to the cloud and DevOps teams started to provision their own infrastructure, security teams that were used to having a controlled environment in local data centers realized their perimeter had widened.Thus, security teams in charge of securing cloud infrastructure need a different approach. They must also quickly adapt to the dynamic nature of the ephemeral infrastructure.

Cloud-bound teams must also quickly adapt to the new paradigm of the cloud infrastructure environment (immutable infra, the policy as code, and identity as the new perimeter, among others).

Like in local data centers, security professionals had to be sure to meet compliance in the hosts instances, user accounts, and data privacy. But the lack of visibility to know what assets they have in the cloud makes it really difficult to keep track of misconfigurations in those assets.

Cloud Security Posture Management (CSPM) is the solution that unifies the different use cases aimed to protect the cloud control plane, basically tracking cloud resources and verifying the static configuration of the cloud. Some CSPM solutions will add extended capabilities, like providing remediation.

Also, one of the main use cases of CSPM is to check that cloud settings are following best practices. Having mapped out-of-the-box frameworks controls and benchmarks can save cloud teams time when addressing things like:

Data storage exposed directly to the internet.
Lack of encryption on databases.
Lack of multi-factor authentication enabled on critical system accounts.

Getting notified if a violation occurs lets teams take action to prioritize its remediation.

CIEM

Identity Management and data privacy are also important aspects of a cloud security program.

As mentioned before, when the perimeter was the local data center, it was easier to control who had access to what. Now, even serverless functions can act like users who access data.

To address the cloud permissions gap, we have Cloud Infrastructure Entitlement Management (CIEM). With CIEM, you would not only know which human and non-human identities can access which resource, but what permission they are using on a daily basis, and suggest policy modifications to enforce least privilege access.

Let’s say we have a group of users who are part of a project. These users are responsible for uploading images into an ECR repository and running those containers in EC2 instances, as well as a number of auto-scaling actions. There’s no need for them to have all the permissions an administrator has, even though that approach may be the simplest to configure. Are they going to be deleting VPCs? That is not one of their tasks. Getting rid of excessive permissions is the first step to reducing collateral damage from credential theft.

Finally CNAPP

If you make it here, congratulations! You are about to uncover the figure after connecting the dots.

We were saying CNAPP is the combination of different use cases that fall into the CWPP, CSPM, and CIEM categories, but let’s go to the source:

“Cloud-native application protection platform (CNAPP) provides more than CWPP-CSPM convergence: There are two important drivers for CNAPP. Firstly, CWPP vendors are looking to posture to provide workload context. Secondly, CSPMs are challenged to provide more and more visibility while “drilling down” into the workload. CNAPP integrates CSPM and CWPP to offer both, and potentially augments them with additional cloud security capabilities.”

Gartner, Inc., How to Protect Your Clouds with CSPM, CWPP, CNAPP, and CASB, 2021, Richard Bartley, May 6, 2021

We hope CNAPP and the rest of the terms make more sense now than when you started reading the article.

Conclusion

CNAPP solutions will promote collaboration between teams (SecDevOps, DevOps, and cloud security operations) by incorporating common workflows, data correlations, meaningful insights, and remediation that’d reduce friction between the personas.

True CNAPP solutions will provide interrelationships between the different insights of the use cases. It’s totally useless to have a nice UI that provides vulnerability scanning if you don’t enrich it with the cloud context of where those images are stored/running. We are not talking about isolated tools put together to call it a day.

Introduction to Adversarial Machine Learning

Miguel — Thu, 09 Dec 2021 13:59:03 +0000

Adversarial machine learning is concerned with the design of ML algorithms that can resist security challenges, the study of the capabilities of attackers, and the understanding of attack consequences.

Adversarial Machine Learning states that there are four types of attacks that ML models can suffer.

Extraction attacks

In a model extraction attack, an adversary steals a copy of a remotely deployed machine learning model, given oracle prediction access.

It is produced by making requests to the target model with inputs to extract as much information as possible and with the set of inputs and outputs train a model called substitute model.

Extract model is hard, the attacker needs a huge compute capacity to re-training the new model with accuracy and fidelity, and substitute model is equivalen to training a model from the ground up.

Defenses

Limit the output information when the model classifies a given input.
Differential Privacy.
Use ensembles.
Proxy between end-user and model like PRADA.
Limit the number of requests.

Inference attacks

Inference attack aim to reverse the information flow of a machine learning model. They allow an adversary to have knowledge of the model that was not explicitly intended to be shared.

Inference attacks pose severe privacy and security threats to individuals and systems. They are successful because private data are statistically correlated with public data, and ML classifiers can capture such statistical correlations.

Includes three types of attacks:

Membership Inference Attack (MIA).
Property Inference Attack (PIA).
Recovery training data.

Defenses

Use advanced cryptography.
- Differential cryptography.
- Homomorphic cryptography.
- Secure Multi-party Computation.
Techniques such as Dropout.
Model compression.

Poisoning attacks

This technique involves an attacker inserting corrupt data in the training dataset to compromise a target machine learning model during training.

Some data poisoning techniques aim to trigger a specific behavior in a computer vision system when it faces a specific pattern of pixels at inference time. Other data poisoning techniques aim to reduce the accuracy of a machine learning model on one or more output classes.

This attack is difficult to detect when performed on training data, since the attack can propagate between different models using the same data.

The adversary seeks to destroy the availability of the model by modifying the decision boundary and, as a result, producing incorrect predictions.

Finally, the attacker could create a backdoor in a model. The model behaves correctly (returning the desired predictions) in most cases, except for certain inputs specially created by the adversary that produce undesired results. The adversary can manipulate the results of the predictions and launch future attacks.

Defenses

Protect the integrity of training data.
Protect the algorithms, use robust methods to train models.

Evasion attacks

An adversary inserts a small perturbation (in the form of noise) into the input of a machine learning model to make it classify incorrectly (example adversary).

They are similar to poisoning attacks, but their main difference is that evasion attacks try to exploit weaknesses of the model in the inference phase, not in the training.

Attacker’s knowledge of the target system is important. The more they know about your model and how its built — the easier it is for them to mount an attack on it.

An evasion attack happens when the network is fed an “adversarial example” — a carefully perturbed input that looks and feels exactly the same as its untampered copy to a human — but that completely throws off the classifier.

Defenses

Training with adversarial examples which robust the model.
Transform the input to the model (Input sanitization).
Gradient regularization.

Tools

Adversarial Robustness Toolbox (ART)

Adversarial Robustness Toolbox (ART) is a Python library for Machine Learning Security. ART provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of Evasion, Poisoning, Extraction, and Inference.

ART supports all popular machine learning frameworks:

TensorFlow
Keras
PyTorch
scikit-learn

All data types:

Images
Tables
Audio
Video

And machine learning tasks:

Classification
Object detection
Speech recognition

pip installation

pip install adversarial-robustness-toolbox

Attack example

from art.attacks.evasion import FastGradientMethod
attack_fgm = FastGradientMethod(estimator = classifier, eps = 0.2)
x_test_fgm = attack_fgm.generate(x=x_test)
predictions_test = classifier.predict(x_test_fgm)

Defense example

from art.defences.trainer import AdversarialTrainer
model.compile(loss=keras.losses.categorical_crossentropy, optimizer=tf.keras.optimizers.Adam(lr=0.01), metrics=["accuracy"])
defence = AdversarialTrainer(classifier=classifier, attacks=attack_fgm, ratio=0.6)
(x_train, y_train), (x_test, y_test), min_pixel_value, max_pixel_value = load_mnist()
defence.fit(x=x_train, y=y_train, nb_epochs=3)

Counterfit

Counterfit is a command-line tool and generic automation layer for assessing the security of machine learning systems.

Developed for security audits on ML models. Implements black box evasion algorithms and based on ART and TextAttack.

Command list

---------------------------------------------------
Microsoft
                          __            _____ __
  _________  __  ______  / /____  _____/ __(_) /_
 / ___/ __ \/ / / / __ \/ __/ _ \/ ___/ /_/ / __/
/ /__/ /_/ / /_/ / / / / /_/  __/ /  / __/ / /
\___/\____/\__,_/_/ /_/\__/\___/_/  /_/ /_/\__/

                                        #ATML

---------------------------------------------------

list targets

list frameworks

load <framework> 

list attacks

interact <target>

predict -i <ind>

use <attack>

run

scan

Final words

"If you use machine learning, there is the risk for exposure, even though the threat does not currently exist in your space." and "The gap between machine learning and security is definitely there." by Hyrum Anderson, Microsoft

References

Thanks

Special thanks to @jiep as a co-writer this article.

GCP security best practices

Miguel — Mon, 29 Nov 2021 18:22:50 +0000

You’ve got a problem to solve and turned to Google Cloud Platform and follow GCP security best practices to build and host your solution. You create your account and are all set to brew some coffee and sit down at your workstation to architect, code, build, and deploy. Except… you aren’t.

There are many knobs you must tweak and practices to put into action if you want your solution to be operative, secure, reliable, performant, and cost effective. First things first, the best time to do that is now – right from the beginning, before you start to design and engineer.

GCP shared responsibility model

The scope of Google Cloud products and services ranges from conventional Infrastructure as a Service (IaaS) to Platform as a Service (PaaS) and Software as a Service (SaaS). As shown in the figure, the traditional boundaries of responsibility between users and cloud providers change based on the service they choose.

At the very least, as part of their common responsibility for security, public cloud providers need to be able to provide you with a solid and secure foundation. Also, providers need to empower you to understand and implement your own parts of the shared responsibility model.

Initial setup GCP security best practices

First, a word of caution: Never use a non-corporate account.

Instead, use a fully managed corporate Google account to improve visibility, auditing, and control of access to Cloud Platform resources. Don’t use email accounts outside of your organization, such as personal accounts, for business purposes.

Cloud Identity is a stand-alone Identity-as-a-Service (IDaaS) that gives Google Cloud users access to many of the identity management features that Google Workspace provides. It is a suite of secure cloud-native collaboration and productivity applications from Google. Through the Cloud Identity management layer, you can enable or disable access to various Google solutions for members of your organization, including Google Cloud Platform (GCP).

Signing up for Cloud Identity also creates an organizational node for your domain. This helps you map your corporate structure and controls to Google Cloud resources through the Google Cloud resource hierarchy.

Now, activating Multi-Factor Authentication (MFA) is the most important thing you want to do. Do this for every user account you create in your system if you want to have a security-first mindset, especially crucial for administrators. MFA, along with strong passwords, are the most effective way to secure user’s accounts against improper access.

Now that you are set, let’s dig into the GCP security best practices.

Identity and Access Management (IAM)

GCP Identity and Access Management (IAM) helps enforce least privilege access control to your cloud resources. You can use IAM to restrict who is authenticated (signed in) and authorized (has permissions) to use resources.

1. Ensure that MFA is enabled for all user accounts

Multi-factor authentication requires more than one mechanism to authenticate a user. This secures user logins from attackers exploiting stolen or weak credentials. By default, multi-factor authentication is not set.

Make sure that for each Google Cloud Platform project, folder, or organization, multi-factor authentication for each account is set and, if not, set it up.

2. Ensure Security Key enforcement for admin accounts

GCP users with Organization Administrator roles have the highest level of privilege in the organization.

These accounts should be protected with the strongest form of two-factor authentication: Security Key Enforcement. Ensure that admins use Security Keys to log in instead of weaker second factors, like SMS or one-time passwords (OTP). Security Keys are actual physical keys used to access Google Organization Administrator Accounts. They send an encrypted signature rather than a code, ensuring that logins cannot be phished.

Identify users with Organization Administrator privileges:

gcloud organizations get-iam-policy ORGANIZATION\_ID

Look for members granted the role ”roles/resourcemanager.organizationAdmin” and then manually verify that Security Key Enforcement has been enabled for each account. If not enabled, take it seriously and enable it immediately. By default, Security Key Enforcement is not enabled for Organization Administrators.

If an organization administrator loses access to their security key, the user may not be able to access their account. For this reason, it is important to configure backup security keys.

Other GCP security IAM best practices include:

Service accounts should not have Admin privileges.
IAM users should not be assigned the Service Account User or Service Account Token Creator roles at project level.
User-managed / external keys for service accounts should be rotated every 90 days or less.
Separation of duties should be enforced while assigning service account related roles to users.
Separation of duties should be enforced while assigning KMS related roles to users.
API keys should not be created for a project.
API keys should be restricted to use by only specified Hosts and Apps.
API keys should be restricted to only APIs that the application needs access to.
API keys should be rotated every 90 days or less.

Key Management Service (KMS)

GCP Cloud Key Management Service (KMS) is a cloud-hosted key management service that allows you to manage symmetric and asymmetric encryption keys for your cloud services in the same way as onprem. It lets you create, use, rotate, and destroy AES 256, RSA 2048, RSA 3072, RSA 4096, EC P256, and EC P384 encryption keys.

3. Check for anonymously or publicly accessible Cloud KMS keys

Anyone can access the dataset by granting permissions to allUsers or allAuthenticatedUsers. Such access may not be desirable if sensitive data is stored in that location.

In this case, make sure that anonymous and/or public access to a Cloud KMS encryption key is not allowed. By default, Cloud KMS does not allow access to allUsers or allAuthenticatedUsers.

List all Cloud KMS keys:

gcloud kms keys list --keyring=KEY\_RING\_NAME --location=global --format=json | jq '.\[\].name'

Remove IAM policy binding for a KMS key to remove access to allUsers and allAuthenticatedUsers:

gcloud kms keys remove-iam-policy-binding KEY\_NAME --keyring=KEY\_RING\_NAME --location=global --member=allUsers --role=ROLE
gcloud kms keys remove-iam-policy-binding KEY\_NAME --keyring=KEY\_RING\_NAME --location=global --member=allAuthenticatedUsers --role=ROLE

The following is a Cloud Custodian rule for detecting the existence of anonymously or publicly accessible Cloud KMS keys:

\- name: anonymously-or-publicly-accessible-cloud-kms-keys
  description: |
    It is recommended that the IAM policy on Cloud KMS cryptokeys should
    restrict anonymous and/or public access.
  resource: gcp.kms-cryptokey
  filters:
    - type: iam-policy
      key: "bindings\[\*\].members\[\]"
      op: intersect
      value: \["allUsers", "allAuthenticatedUsers"\]

Cloud Storage

Google Cloud Storage lets you store any amount of data in namespaces called “buckets.” These buckets are an appealing target for any attacker who wants to get hold of your data, so you must take great care in securing them.

4. Ensure that Cloud Storage buckets are not anonymously or publicly accessible

Allowing anonymous or public access gives everyone permission to access bucket content. Such access may not be desirable if you are storing sensitive data. Therefore, make sure that anonymous or public access to the bucket is not allowed.

List all buckets in a project:

gsutil ls

Check the IAM Policy for each bucket returned from the above command:

gsutil iam get gs://BUCKET\_NAME

No role should contain allUsers or allAuthenticatedUsers as a member. If that’s not the case, you’ll want to remove them with:

gsutil iam ch -d allUsers gs://BUCKET\_NAME
gsutil iam ch -d allAuthenticatedUsers gs://BUCKET\_NAME

Also, you might want to prevent Storage buckets from becoming publicly accessible by setting up the Domain restricted sharing organization policy.

Compute Engine

Compunte Engine provides security and customizable compute service that lets you create and run virtual machines on Google’s infrastructure.

5. Ensure VM disks for critical VMs are encrypted with Customer-Supplied Encryption Keys (CSEK)

By default, the Compute Engine service encrypts all data at rest.

Cloud services manage this type of encryption without any additional action from users or applications. However, if you want full control over instance disk encryption, you can provide your own encryption key.

These custom keys, also known as Customer-Supplied Encryption Keys (CSEKs), are used by Google Compute Engine to protect the Google-generated keys used to encrypt and decrypt instance data. The Compute Engine service does not store CSEK on the server and cannot access protected data unless you specify the required key.

At the very least, business critical VMs should have VM disks encrypted with CSEK.

By default, VM disks are encrypted with Google-managed keys. They are not encrypted with Customer-Supplied Encryption Keys.

Currently, there is no way to update the encryption of an existing disk, so you should create a new disk with Encryption set to Customer supplied. A word of caution is necessary here:

⚠️ If you lose your encryption key, you will not be able to recover the data.

In the gcloud compute tool, encrypt a disk using the --csek-key-file flag during instance creation. If you are using an RSA-wrapped key, use the gcloud beta component:

gcloud beta compute instances create INSTANCE\_NAME --csek-key-file=key-file.json

To encrypt a standalone persistent disk use:

gcloud beta compute disks create DISK\_NAME --csek-key-file=key-file.json

It is your duty to generate and manage your key. You must provide a key that is a 256-bit string encoded in RFC 4648 standard base64 to the Compute Engine. A sample key-file.json looks like this:

\[
  {
    "uri": "https://www.googleapis.com/compute/v1/projects/myproject/zones/us-
central1-a/disks/example-disk",
    "key": "acXTX3rxrKAFTF0tYVLvydU1riRZTvUNC4g5I11NY-c=",
    "key-type": "raw"
  },
  {
    "uri":
"https://www.googleapis.com/compute/v1/projects/myproject/global/snapshots/my
-private-snapshot",
    "key":
"ieCx/NcW06PcT7Ep1X6LUTc/hLvUDYyzSZPPVCVPTVEohpeHASqC8uw5TzyO9U+Fka9JFHz0mBib
XUInrC/jEk014kCK/NPjYgEMOyssZ4ZINPKxlUh2zn1bV+MCaTICrdmuSBTWlUUiFoDD6PYznLwh8
ZNdaheCeZ8ewEXgFQ8V+sDroLaN3Xs3MDTXQEMMoNUXMCZEIpg9Vtp9x2oeQ5lAbtt7bYAAHf5l+g
JWw3sUfs0/Glw5fpdjT8Uggrr+RMZezGrltJEF293rvTIjWOEB3z5OHyHwQkvdrPDFcTqsLfh+8Hr
8g+mf+7zVPEC8nEbqpdl3GPv3A7AwpFp7MA=="
    "key-type": "rsa-encrypted"
  }
\]

Other GCP security best practices for Compute Engine include:

Ensure that instances are not configured to use the default service account.
Ensure that instances are not configured to use the default service account with full access to all Cloud APIs.
Ensure oslogin is enabled for a Project.
Ensure that IP forwarding is not enabled on Instances.
Ensure Compute instances are launched with Shielded VM enabled.
Ensure that Compute instances do not have public IP addresses.
Ensure that App Engine applications enforce HTTPS connections.

Google Kubernetes Engine Service (GKE)

The Google Kubernetes Engine (GKE) provides a managed environment for deploying, managing, and scaling containerized applications using the Google infrastructure. A GKE environment consists of multiple machines (specifically, Compute Engine instances) grouped together to form a cluster.

6. Enable application-layer secrets encryption for GKE clusters

Application-layer secret encryption provides an additional layer of security for sensitive data, such as Kubernetes secrets stored on etcd. This feature allows you to use Cloud KMS managed encryption keys to encrypt data at the application layer and protect it from attackers accessing offline copies of etcd. Enabling application-layer secret encryption in a GKE cluster is considered a security best practice for applications that store sensitive data.

Create a key ring to store the CMK:

gcloud kms keyrings create KEY\_RING\_NAME --location=REGION --project=PROJECT\_NAME --format="table(name)"

Now, create a new Cloud KMS Customer-Managed Key (CMK) within the KMS key ring created at the previous step:

gcloud kms keys create KEY\_NAME --location=REGION --keyring=KEY\_RING\_NAME --purpose=encryption --protection-level=software --rotation-period=90d --format="table(name)"

And lastly, assign the Cloud KMS “CryptoKey Encrypter/Decrypter” role to the appropriate service account:

gcloud projects add-iam-policy-binding PROJECT\_ID --member=serviceAccount:service-PROJECT\_NUMBER@container-engine-robot.iam.gserviceaccount.com --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

The final step is to enable application-layer secrets encryption for the selected cluster, using the Cloud KMS Customer-Managed Key (CMK) created in the previous steps:

gcloud container clusters update CLUSTER --region=REGION --project=PROJECT\_NAME --database-encryption-key=projects/PROJECT\_NAME/locations/REGION/keyRings/KEY\_RING\_NAME/cryptoKeys/KEY\_NAME

7. Enable GKE cluster node encryption with customer-managed keys

To give you more control over the GKE data encryption / decryption process, make sure your Google Kubernetes Engine (GKE) cluster node is encrypted with a customer-managed key (CMK). You can use the Cloud Key Management Service (Cloud KMS) to create and manage your own customer-managed keys (CMKs). Cloud KMS provides secure and efficient cryptographic key management, controlled key rotation, and revocation mechanisms.

At this point, you should already have a key ring where you store the CMKs, as well as customer-managed keys. You will use them here too.

To enable GKE cluster node encryption, you will need to re-create the node pool. For this, use the name of the cluster node pool that you want to re-create as an identifier parameter and custom output filtering to describe the configuration information available for the selected node pool:

gcloud container node-pools describe NODE\_POOL --cluster=CLUSTER\_NAME --region=REGION --format=json

Now, using the information returned in the previous step, create a new Google Cloud GKE cluster node pool, encrypted with your customer-managed key (CMK):

gcloud beta container node-pools create NODE\_POOL --cluster=CLUSTER\_NAME --region=REGION --disk-type=pd-standard --disk-size=150 --boot-disk-kms-key=projects/PROJECT/locations/REGION/keyRings/KEY\_RING\_NAME/cryptoKeys/KEY\_NAME

Once your new cluster node pool is working properly, you can delete the original node pool to stop adding invoices to your Google Cloud account.

⚠️ Take good care to delete the old pool and not the new one!

gcloud container node-pools delete NODE\_POOL --cluster=CLUSTER\_NAME --region=REGION

8. Restrict network access to GKE clusters

To limit your exposure to the Internet, make sure your Google Kubernetes Engine (GKE) cluster is configured with a master authorized network. Master authorized networks allow you to whitelist specific IP addresses and/or IP address ranges to access cluster master endpoints using HTTPS.

Adding a master authorized network can provide network-level protection and additional security benefits to your GKE cluster. Authorized networks allow access to a particular set of trusted IP addresses, such as those originating from a secure network. This helps protect access to the GKE cluster if the cluster’s authentication or authorization mechanism is vulnerable.

Add authorized networks to the selected GKE cluster to grant access to the cluster master from the trusted IP addresses / IP ranges that you define:

gcloud container clusters update CLUSTER\_NAME --zone=REGION --enable-master-authorized-networks --master-authorized-networks=CIDR\_1,CIDR\_2,...

In the previous command, you can specify multiple CIDRs (up to 50) separated by a comma.

The above are the most important best practices for GKE, since not adhering to them poses a high risk, but there are other security best practices you might want to adhere to:

Enable auto-repair for GKE cluster nodes.
Enable auto-upgrade for GKE cluster nodes.
Enable integrity monitoring for GKE cluster nodes.
Enable secure boot for GKE cluster nodes.
Use shielded GKE cluster nodes.

Cloud Logging

Cloud Logging is a fully managed service that allows you to store, search, analyze, monitor, and alert log data and events from Google Cloud and Amazon Web Services. You can collect log data from over 150 popular application components, onprem systems, and hybrid cloud systems.

9. Ensure that Cloud Audit Logging is configured properly across all services and all users from a project

Cloud Audit Logging maintains two audit logs for each project, folder, and organization:

Admin Activity and Data Access. Admin Activity logs contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. These are enabled for all services and cannot be configured. On the other hand, Data Access audit logs record API calls that create, modify, or read user-provided data. These are disabled by default and should be enabled.

It is recommended to have an effective default audit config configured in such a way that you can log user activity tracking, as well as changes (tampering) to user data. Logs should be captured for all users.

For this, you will need to edit the project’s policy. First, download it as a yaml file:

gcloud projects get-iam-policy PROJECT\_ID > /tmp/project\_policy.yaml

Now, edit /tmp/project_policy.yaml adding or changing only the audit logs configuration to the following:

auditConfigs:
- auditLogConfigs:
  - logType: DATA\_WRITE
  - logType: DATA\_READ
  service: allServices

Please note that exemptedMembers is not set as audit logging should be enabled for all the users. Last, update the policy with the new changes:

gcloud projects set-iam-policy PROJECT\_ID /tmp/project\_policy.yaml

⚠️ Enabling the Data Access audit logs might result in your project being charged for the additional logs usage.

10. Enable logs router encryption with customer-managed keys

Make sure your Google Cloud Logs Router data is encrypted with a customer-managed key (CMK) to give you complete control over the data encryption and decryption process, as well as to meet your compliance requirements.

You will want to add a policy, binding to the IAM policy of the CMK, to assign the Cloud KMS “CryptoKey Encrypter/Decrypter” role to the necessary service account.

gcloud kms keys add-iam-policy-binding KEY\_ID --keyring=KEY\_RING\_NAME --location=global --member=serviceAccount:PROJECT\_NUMBER@gcp-sa-logging.iam.gserviceaccount.com --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

Cloud SQL

Cloud SQL is a fully managed relational database service for MySQL, PostgreSQL, and SQL Server. Run the same relational databases you know with their rich extension collections, configuration flags and developer ecosystem, but without the hassle of self management.

11. Ensure that Cloud SQL database instances are not open to the world

Only trusted / known required IPs should be whitelisted to connect in order to minimize the attack surface of the database server instance. The allowed networks must not have an IP / network configured to 0.0.0.0/0 that allows access to the instance from anywhere in the world. Note that allowed networks apply only to instances with public IPs.

gcloud sql instances patch INSTANCE_NAME --authorized-networks=IP_ADDR1,IP_ADDR2...

To prevent new SQL instances from being configured to accept incoming connections from any IP addresses, set up a Restrict Authorized Networks on Cloud SQL instances Organization Policy.

BigQuery

BigQuery is a serverless, highly-scalable, and cost-effective cloud data warehouse with an in-memory BI Engine and machine learning built in.

12. Ensure that BigQuery datasets are not anonymously or publicly accessible

You don’t want to allow anonymous or public access in your BigQuery dataset’s IAM policies. Anyone can access the dataset by granting permissions to allUsers or allAuthenticatedUsers. Such access may not be desirable if sensitive data is stored on the dataset. Therefore, make sure that anonymous and/or public access to the dataset is not allowed.

To do this, you will need to edit the data set information. First you need to retrieve said information into your local filesystem:

bq show --format=prettyjson PROJECT\_ID:DATASET\_NAME > dataset\_info.json

Now, in the access section of dataset_info.json, update the dataset information to remove all roles containing allUsers or allAuthenticatedUsers.

Finally, update the dataset:

bq update --source=dataset\_info.json PROJECT\_ID:DATASET\_NAME

You can prevent BigQuery dataset from becoming publicly accessible by setting up the Domain restricted sharing organization policy.

Compliance Standards & Benchmarks

Setting up all the detection rules and maintaining your GCP environment to keep it secure is an ongoing effort that can take a big chunk of your time – even more so if you don’t have some kind of roadmap to guide you during this continuous work.

You will be better off following the compliance standard(s) relevant to your industry, since they provide all the requirements needed to effectively secure your cloud environment.

Because of the ongoing nature of securing your infrastructure and complying with a security standard, you might also want to recurrently run benchmarks, such as CIS Google Cloud Platform Foundation Benchmark, which will audit your system and report any unconformity it might find.

Conclusion

Jumping to the cloud opens a new world of possibilities, but it also requires learning a new set of Google Cloud Platform security best practices.

Each new cloud service you leverage has its own set of potential dangers you need to be aware of.

Luckily, cloud native security tools like Falco and Cloud Custodian can guide you through these Google Cloud Platform security best practices, and help you meet your compliance requirements.

Container security best practices: Comprehensive guide

Miguel — Tue, 16 Nov 2021 13:49:32 +0000

Sticking to container security best practices is critical for successfully delivering verified software, as well as preventing severe security breaches and its consequences.

According to the 2020 CNFC Survey, 92 percent of companies are using containers in production, a 300 percent increase since 2016. Thus, Kubernetes, Openshift, and other container technologies are present everywhere.

But aren’t containers meant to be safe and isolated? Well, kind of.

For example, an exploitable vulnerability inside a container, combined with exposed metadata and a wrong credentials configuration, can compromise your whole cloud infrastructure. As described in our Cloud lateral movement post, a hacker can use this chain of exploits and wrong configurations to run crypto mining applications in your cloud account.

Containers were designed as a distribution mechanism for self-contained applications, allowing them to execute processes in an isolated environment. For isolation purposes, they employ a lightweight mechanism using kernel namespaces, removing the requirement of several additional layers in VMs, like a full operating system, CPU and hardware virtualization, etc.

The lack of these additional abstraction layers, as well as tightly coupling with the kernel, operating system, and container runtime, make it easier to use exploits to jump from inside the container to the outside and vice versa.

Container security best practices don’t just include the delivered applications and the container image itself, but also the full component stack used for building, distributing, and specially executing the container.

This includes:

The host or VM
The container runtime
Cluster technology
Cloud provider configuration
And more.

Security can be applied at each of the different phases: development, distribution, execution, detection and response to threats.

Let’s dive into the interesting details, breaking down the general ideas into 18 concrete container security best practices that you can apply in your DevOps workflows.

A complex stack

Containers’ success is often fueled by two really useful features:

They are a really convenient way to distribute and execute software, as a self-contained executable image which includes all libraries and dependencies, while being much lighter than classical VM images.
They offer a good level of security and isolation by using kernel namespaces to execute processes in their own “jail”, including mounts, PID, network, IPC, etc., and also resource limiting CPU usage and memory via kernel cgroups. Memory protection, permission enforcement, etc. are still provided via the standard kernel security mechanisms.

The container security model might be enough in most cases, but for example, AWS adds additional security for their serverless solution. It does so by running containers inside Firecracker, a micro virtual machine that adds another level of virtualization to prevent cross-customer breaches.

Does this mean containers are not safe?

You can see it as a double-edged sword.

An application running inside a container is no different than an application running directly in a machine, sharing a file-system and processes with many other applications. In a sense, they are just applications that could contain exploitable vulnerabilities.

Running inside a container won’t prevent this, but will make it much harder to jump from the application exploit to the host system, or access data from other applications.

On the other hand, containers depend on another set of kernel features, a container runtime, and usually a cluster or orchestrator that might be exploited too.

So, we need to take the whole stack into account, and we can apply container security best practices at the different phases of the container lifecycle.

There will be cases like the serverless compute engine ECS Fargate, Google Cloud Run, etc., where some of these pieces are out of our control, so we work on a shared responsibility model.

The provider is responsible for keeping the base pieces working and secured
And you can focus on the upper layers.

Prevention: 8 steps for shift left security

Before your application inside a container is executed, there are several places where you can start applying different techniques to prevent threats from happening.

Prevention and applying security as early as possible is key and will save you a lot of trouble, time, and money with minimal effort if you apply some good practices during the development and distribution of the container images.

1. Integrate Code Scanning at the CI/CD Process

Security scanning is the process of analyzing your software, configuration or infrastructure, and detecting potential issues or known vulnerabilities. Scanning can be done at different stages:

Code
Dependencies
Infrastructure as code
Container Images
Hosts
Cloud configuration
… and more

Let’s focus on the first stage: code. Before you ship the application or even build your application, you can scan your code to detect bugs or potentially exploitable code (a new vulnerability).

For application code, there are different SAST (Static Application Security Testing) tools like sonarqube, which provide vulnerability scanners for different languages, gosec for analyzing go code and detecting issues based on rules, linters, etc.

You can run them at the developer machine, but integrating code scanning tools at the CI/CD process can make sure that a minimum level of code quality is assured. For example, you can block pull requests by default if some checks are failing.

A Github Action running gosec:

name: "Security Scan"
on:
  push:
jobs:
  tests:
    runs-on: ubuntu-latest
    env:
      GO111MODULE: on
    steps:
      - name: Checkout Source
        uses: actions/checkout@v2
      - name: Run Gosec Security Scanner
        uses: securego/gosec@master
        with:
          args: ./...

And the corresponding output:

2. Reduce external vulnerabilities via dependency scanning

Only very minimal and toy applications are free of third-party libraries or frameworks. But reusing code from external dependencies means you will be including bugs and vulnerabilities from these dependencies as part of your application. Dependency scanning should be included as a best practice in any application build process.

Package management tools, like npm, maven, go, etc., can match vulnerability databases with your application dependencies and provide useful warning.

For example, enabling the dependency-check plugin in Maven requires just adding a plugin to the pom.xml:

<project>
    ...
    <build>
        ...
        <plugins>
            ...
            <plugin>
              <groupId>org.owasp</groupId>
              <artifactId>dependency-check-maven</artifactId>
              <version>6.2.2</version>
              <executions>
                  <execution>
                      <goals>
                          <goal>check</goal>
                      </goals>
                  </execution>
              </executions>
            </plugin>
            ...
        </plugins>
        ...
    </build>
    ...
</project>

And every time maven is executed, it will generate a vulnerability report:

Avoid introducing vulnerabilities through dependencies by updating them to newer versions with fixes.

In some cases, this might not be possible because the fix is not available, or bumping the version would require a lot of refactoring due to breaking changes. Analyze the vulnerabilities revealed by dependency scanning to evaluate the impact and exploitability, and introduce additional measures like checks in your code or protection mechanisms to prevent the vulnerability from being exploited.

Note that although it is possible to also scan dependencies later, once the application is built, dependency scanning will be less accurate as some metadata information is not available, and it might be impossible for statically linked applications like Go or Rust.

3. Use image scanning to analyze container images

Once your application is built and packaged, it is common to copy it inside a container with a minimal set of libraries, dependent frameworks (like Python, Node, etc.), and configuration files. You can read our Top 20 Dockerfile best practices to learn about the best practices focused in container building and runtime.

Use an image scanner to analyze your container images. The image scanning tool will discover vulnerabilities in the operating system packages (rpm, dpkg, apk, etc.) provided by the container image base distribution. It will also reveal vulnerabilities in package dependencies for Java, Node, Python, and others, even if you didn’t apply dependency scanning in the previous stages.

Image Scanning is easy to automate and enforce. It can be included as part of your CI/CD pipelines, triggered when new images are pushed to a registry, or verified in a cluster admission controller to make sure that non-compliant images are now allowed to run. Another option is installing Sysdig Node Image Analyzer to scan images as soon as they start running in the hosts where it is running.

An example is Github Action integration with the Sysdig Secure Inline Scan Action:

name: "Security Scan"
on:
  push:
jobs:
  build-and-scan:
    runs-on: ubuntu-latest
    steps:
    - name: Build the Docker image
      run: docker build . --file Dockerfile --tag my-image:latest
    - name: Scan image
      id: scan
      uses: sysdiglabs/scan-action@v3
      with:
        image-tag: my-image:latest
        sysdig-secure-token: ${{ secrets.SYSDIG\_SECURE\_TOKEN }}
        input-type: docker-daemon
        run-as-user: root

The previous example builds a Docker image and then scans it locally, from the Docker daemon.

Scan results are provided directly as part of the action output, and pull-request can be blocked from merging depending on the check status:

4. Enforce image content trust

Container image integrity can be enforced by adding digital signatures via Docker Notary or similar, which then can be verified in the Admission Controller or the container runtime.

Let’s see a quick example:

$ docker trust key generate example1
Generating key for example1...
Enter passphrase for new example1 key with ID 7d7b320:
Repeat passphrase for new example1 key with ID 7d7b320:
Successfully generated and loaded private key. Corresponding public key available: /Users/airadier/example1.pub

Now, we have a signing key called “example1”. The public part is located in:

$HOME/example1.pub

and the private counterpart will be located in:

$HOME/.docker/trust/private/<key ID>.key

Other developers can also generate their keys and share the public part.

Now, we enable a signed repository by adding the keys of the allowed signers to the repository (airadier/alpine in the example):

$ docker trust signer add --key example1.pub example1 airadier/alpine
Adding signer "example1" to airadier/alpine...
Initializing signed repository for airadier/alpine...
...
Enter passphrase for new repository key with ID 16db658:
Repeat passphrase for new repository key with ID 16db658:
Successfully initialized "airadier/alpine"
Successfully added signer: example1 to airadier/alpine

And we can sign an image in the repository with:

$ docker trust sign airadier/alpine:latest
Signing and pushing trust data for local image airadier/alpine:latest, may overwrite remote trust data
The push refers to repository \[docker.io/airadier/alpine\]
bc276c40b172: Layer already exists
latest: digest: sha256:be9bdc0ef8e96dbc428dc189b31e2e3b05523d96d12ed627c37aa2936653258c size: 528
Signing and pushing trust metadata
Enter passphrase for example1 key with ID 7d7b320:
Successfully signed docker.io/airadier/alpine:latest

If the DOCKER_CONTENT_TRUST environment variable is set to 1, then pushed images will be automatically signed:

$ export DOCKER\_CONTENT\_TRUST=1
$ docker push airadier/alpine:3.11
The push refers to repository \[docker.io/airadier/alpine\]
3e207b409db3: Layer already exists
3.11: digest: sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 size: 528
Signing and pushing trust metadata
Enter passphrase for example1 key with ID 7d7b320:
Successfully signed docker.io/airadier/alpine:3.11

We can check the signers of an image with:

$ docker trust inspect --pretty airadier/alpine:latest
Signatures for airadier/alpine:latest
SIGNED TAG   DIGEST                                                             SIGNERS
latest       be9bdc0ef8e96dbc428dc189b31e2e3b05523d9...   example1
List of signers and their keys for airadier/alpine:latest
SIGNER     KEYS
example1   7d7b320791b7
Administrative keys for airadier/alpine:latest
  Repository Key:       16db658159255bf0196...
  Root Key:             2308d2a487a1f2d499f184ba...

When the environment variable DOCKER_CONTENT_TRUST is set to 1, the Docker CLI will refuse to pull images without trust information:

$ export DOCKER\_CONTENT\_TRUST=1
$ docker pull airadier/alpine-ro:latest
Error: remote trust data does not exist for docker.io/airadier/alpine-ro: notary.docker.io does not have trust data for docker.io/airadier/alpine-ro

You can enforce content trust in a Kubernetes cluster by using an Admission controller like Connaisseur.

5. Common security misconfigurations and remediations

Wrongly configured hosts, container runtimes, clusters, or cloud resources can leave a door open to an attack, or create an easy way to escalate privileges and perform lateral movement.

Benchmarks, best practices, and hardening guides provide you with information about how to spot those misconfigurations, why they are a problem, and how to remediate them. Among different sources of information, the Center for Internet Security (CIS) is paramount. It’s a non-profit organization that publishes free benchmarks for many different environments, where any person and company can contribute with their knowledge. It has become a de facto standard for security benchmarking.

The best way to make sure you can check this kind of setting for container security is to automate it as much as possible. Several tools exist for this, mainly based on static configuration analysis, allowing you to check configuration parameters at different levels and provide guidance in fixing them.

Sysdig Secure includes a Compliance and Benchmarks feature which can help you schedule, execute, and analyze all of your infrastructure (Linux hosts, Docker, Kubernetes, EKS, GKE, Openshift clusters, etc.) based on CIS Benchmarks, as well as compliance standards, like PCI DSS, SOC 2, NIST 800-53, NIST 800-190, HIPAA, ISO 27001, GDPR and others, all in a single centralized dashboard.

Other tools you can use are linux-bench, docker-bench, kube-bench, kube-hunter, kube-striker, Cloud Custodian, OVAL, and OS Query.

Example host benchmark control

A physical machine where you just installed Linux, a virtual machine provisioned on a cloud provider, or on-prem may contain several insecure out-of-the-box configurations that you are not aware of. If you plan to use it for a prolonged period, with a production workload or exposure to the internet, you have to take special care of them. This is also true for Kubernetes or OpenShift nodes. After all, they are virtual machines; don’t assume that if you are using a cluster provisioned by your cloud provider that they come perfectly secured.

CIS has a benchmark for Distribution Independant Linux, and one specifically for Debian, CentOs, Red Hat, and many other distributions.

Examples of misconfigurations you can detect:

Example linux distribution

The following figure is provided by CIS Benchmark for Distribution Independant Linux, the configuration is to ensure rsh server is not enabled.

Example container runtime benchmark control

If you install a container runtime like Docker by yourself in a server you own, it’s essential you use a benchmark to make sure any default insecure configuration is remediated. The next figure shows the configuration to ensure that authorization for Docker client commands is enabled.

Example orchestrator benchmark control

Kubernetes, by default, leaves many authentication mechanisms to be managed by third-party integrations. A benchmark will ensure all possible insecurities are dealt with. The image below show us the configuration to ensure that the –anonymous-auth argument is set to false.

Example cloud benchmark control

Benchmarks on cloud provider accounts, also called Cloud Security Posture Management (CSPM), are essential, as they will check the security on every single asset on the account. All settings that could lead to an attack, resources that should be private but are made public (e.g., S3 buckets), or storage that lacks encryption are defined in this kind of benchmark. This is a benchmark that is essential to automate, as the assets in the cloud account change all the time, and you have to constantly watch that everything is as secure as possible. The following image is an example of configuration check that ensures credentials unused for 90 days or greater are disabled.

6. Incorporate IaC scanning

Cloud resource management is a complex task, and tools like Terraform or CloudFormation can help leverage this burden. Infrastructure is declared as code – aka “Infrastructure as Code” – stored and versioned in a repository, and automation takes care of applying the changes in the definition to keep the existing infrastructure up to date with the declaration.

If you are using infrastructure as code, incorporate IaC scanning tools like Apolicy, Checkov, tfsec, or cfn_nag to validate the configuration of your infrastructure before it is created or updated. Similar to other linting tools, apply IaC scanning tools locally and in your pipeline, and consider blocking changes that introduce security issues.

An example of a checkov execution:

$ pip install checkov
$ checkov --quiet -d .
       \_               \_
   \_\_\_| |\_\_   \_\_\_  \_\_\_| | \_\_\_\_\_\_\_   \_\_
  / \_\_| '\_ \\ / \_ \\/ \_\_| |/ / \_ \\ \\ / /
 | (\_\_| | | |  \_\_/ (\_\_|   < (\_) \\ V /
  \\\_\_\_|\_| |\_|\\\_\_\_|\\\_\_\_|\_|\\\_\\\_\_\_/ \\\_/
By bridgecrew.io | version: 2.0.346
terraform scan results:
Passed checks: 314, Failed checks: 57, Skipped checks: 0
Check: CKV\_AWS\_108: "Ensure IAM policies does not allow data exfiltration"
        FAILED for resource: aws\_iam\_policy\_document.cloudtrail\_ingestor
        File: /modules/ingestor/main.tf:17-31
        Guide: https://docs.bridgecrew.io/docs/ensure-iam-policies-do-not-allow-data-exfiltration
                17 | data "aws\_iam\_policy\_document" "ingestor" {
                18 |   statement {
                19 |     effect = "Allow"
                20 |     actions = \[
                21 |       "s3:Get\*",
                22 |       "s3:List\*",
                23 |       "s3:Put\*",
                24 |       "s3:Head\*",
                25 |       "sqs:DeleteMessage",
                26 |       "sqs:DeleteMessageBatch",
                27 |       "sqs:ReceiveMessage",
                28 |     \]
                29 |     resources = \["\*"\]
                30 |   }
                31 | }
...

7. Secure your host with host scanning

Securing your host is just as important as securing the containers. The host where the containers are running is usually composed of an operating system with a Linux kernel, a set of libraries, a container runtime, and other common services and helpers running in the background. Any of these components can be vulnerable or misconfigured, and could be used as the entry point to access the running containers or cause a denial of service attack.

For example, issues in the container runtime itself can cause an impact in your running containers, like this DoS attack that prevents creating new containers in a host.

We already talked about hardening host configuration in the “Unsafe configuration” section. But how do we detect vulnerable components? A host scanning tool can detect known vulnerabilities in the kernel, standard libraries like glibc, services, and even the container runtime living in the host (quite similar to what image scanning does for a container image).

Sysdig Host Analyzer will transparently scan your hosts and report found vulnerabilities, The following figure shows how easy it is to detect risks at a glance on the dashboard.

Use this information to update the operating system, kernel, packages, etc. Get rid of the most critical and exploitable vulnerabilities, or at least be aware of them, and apply other protection mechanisms like firewalls, restricting user access to the host, stopping unused services, etc.

8. Prevent unsafe containers from running

As a last line of defense, Kubernetes Admission Controllers can block unsafe containers from running in the cluster.

Sysdig Admission Controller allows you to deny the creation of pods running images that don’t pass your security policies, based on the scanning results.

Gatekeeper provides a powerful language that can be used to define flexible rules to accept or reject containers based on the pod specification (e.g., enforce annotations, detect privileged pods, or using host paths) and the status of the cluster (e.g.m, require all ingress hosts to be unique within the cluster).

As an example, the following Gatekeeper ConstrainsTemplate (some data is ellipsed) defines a template for detecting required annotations:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredannotations
  annotations:
    description: Requires all resources to contain a specified annotation(s) with a value
      matching a provided regular expression.
spec:
 ...
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredannotations
        violation\[{"msg": msg, "details": {"missing\_annotations": missing}}\] {
            provided := {annotation | input.review.object.metadata.annotations\[annotation\]}
            required := {annotation | annotation := input.parameters.annotations\[\_\].key}
            missing := required - provided
            count(missing) > 0
            msg := sprintf("you must provide annotation(s): %v", \[missing\])
        }
        violation\[{"msg": msg}\] {
          value := input.review.object.metadata.annotations\[key\]
          expected := input.parameters.annotations\[\_\]
          expected.key == key
          expected.allowedRegex != ""
          not re\_match(expected.allowedRegex, value)
          msg := sprintf("Annotation <%v: %v> does not satisfy allowed regex: %v", \[key, value, expected.allowedRegex\])
        }

Using that template, we can enforce that all services have some annotations with:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredAnnotations
metadata:
  name: all-must-have-certain-set-of-annotations
spec:
  match:
    kinds:
      - apiGroups: \[""\]
        kinds: \["Service"\]
  parameters:
    message: "All services must have a \`a8r.io/owner\` and \`a8r.io/runbook\` annotations."
    annotations:
      - key: a8r.io/owner
        # Matches email address or github user
        allowedRegex: ^(\[A-Za-z0-9.\_%+-\]+@\[A-Za-z0-9.-\]+\\.\[A-Za-z\]{2,6}|\[a-z\]{1,39})$
      - key: a8r.io/runbook
        # Matches urls including or not http/https
        allowedRegex: ^(http:\\/\\/www\\.|https:\\/\\/www\\.|http:\\/\\/|https:\\/\\/)?\[a-z0-9\]+(\[\\-\\.\]{1}\[a-z0-9\]+)\*\\.\[a-z\]{2,5}(:\[0-9\]{1,5})?(\\/.\*)?$

Many more examples are available in the OPA Gatekeeper library project!

We already mentioned Connaisseur Admission Controller as a way to enforce content trust and reject images that are not signed by trusted sources.

Protection – Running your containers safely

Adhering to build time and configuration container security best practices right before runtime still won’t make your container 100 percent safe. New container vulnerabilities are discovered daily, so your actual container, quite safe today, can become a potential victim of new disclosed exploits tomorrow.

In this section, we will introduce container security best practices for including container vulnerability management and protection measures in your workload.

9. Protect your resources

Your containers and host might contain vulnerabilities, and new ones are discovered continually. However, the danger is not in the host or container vulnerability itself, but rather the attack vector and exploitability.

For example, you can protect from a network exploitable vulnerability by impeding connections to the running container or the vulnerable service. And if the attack vector requires local access to the host (being logged in the host), you can restrict the access to that host.

So, limit the number of users that have access to your hosts, cloud accounts, and resources, and block unnecessary network traffic by using different mechanisms:

VPCs, Security groups, network rules, firewall rules, etc. in cloud providers to restrict communication between VMs, VPCs, and the Internet.
Firewalls at hosts levels to expose only the minimal set of required services.
Kubernetes Network Policies for clusters, and additional tools, like Service Mesh or API Gateways, can add an additional security layer for filtering network requests.

10. Verify image signatures

As described in “Image content trust”, image signatures are a protection mechanism to guarantee that the image has not been tampered. Verifying image signatures can also prevent some attacks with tag mutability, assuring that the tag corresponds to a specific digest that has been signed by the publisher. The figure below shows an example of this attack.

11. Restrict container privileges at runtime

The scope or “blast radius” of an exploited vulnerability inside a container largely depends on the privileges of the container, and the level of isolation from the host and other resources. Runtime configuration can mitigate the impact of existing and future vulnerabilities in the following ways:

Effective user: Don’t run the container as root. Even better, use randomized UIDs (like Openshift) that don’t map to real users in the host, or use the user namespace feature in Docker and in Kubernetes when ready (not available at time of publish).
Restrict container privileges: Docker and Kubernetes offer ways to drop capabilities and don’t allow privileged containers. Seccomp and AppArmor can add more restrictions to the range of actions a container can perform.
Add resource limits: Avoid container consuming all the memory or CPUs and starve other applications.
Be careful with shared storage or volumes: Specifically, things like hostPath, and sharing the filesystem from the host.
Other options like hostNetwork, hostPID or hostIPC: Kubernetes will make the container share a namespace with the host, reducing isolation.
Define Pod Security Policies (PSPs) and Security Context Constraints (SCCs for Openshift): Set guardrails in your cluster and prevent misconfigured containers. PSPs and SCCs are Admission Controllers that will reject pods in case their security context does not comply with the defined policies.

12. Manage container vulnerabilities wisely

Manage and assess your vulnerabilities wisely. Not all vulnerabilities have fixes available, or may now be able to be applied easily.

However, not all of them might be easily exploitable, or they may require local or even physical access to the hosts to be exploited.

You need to have a good strategy, including:

Prioritize what needs to be fixed: You should focus on host and container vulnerabilities with higher score or severity, which often means that they are remotely exploitable and that a public exploit is available. If they are old and well known, the chances are high that they are being actively exploited in an automated way.
Evaluate the severity of vulnerabilities in your environment: The score or severity provided by the vendor or your linux distribution is a good starting point. But a vulnerability can have a high score if it is remotely exploitable, and then exist in an unused package in an internal host which is not exposed to the internet. And it can be in a production environment, or a developers’ playground and experimental cluster. Evaluate the impact in context, and plan accordingly.
Plan applying fixes as countermeasures to protect your containers and hosts: Create and track tickets, making vulnerability management part of your standard development workflows.
Create exceptions for vulnerabilities when you conclude that you are not impacted: This will reduce the noise. Consider snoozing instead of permanently adding an exception, so you can reevaluate later.

Your strategy should translate in policies that a container vulnerability scanner can use to trigger alerts for detected vulnerabilities according to some criteria, and to apply prevention and protection at different levels, like:

Ticketing: Notify developers of detected vulnerabilities so they can apply the fixes.
Image registry: Prevent vulnerable images from being pulled at all.
Host / kernel / container: Block running containers, add additional protection measures or respond by killing, quarantine or shut down hosts or containers for critical issues.

It is also important to perform continuous vulnerability scanning and reevaluation to make sure that you get alerts when new vulnerabilities that apply to running containers are discovered. Sysdig Secure can help here, as it will reevaluate your scanning policies every time the vulnerability feeds are updated.

Detection – Alerts for abnormal behavior

So far, we put the focus on prevention and protection, getting your containers up and running in the best possible shape, and anticipating potential and known attacks. Applying prevention techniques when building, distributing, and running your container with the correct privileges and protections, as well as ensuring the underlying stack, will limit the range of action that an attacker can take. But that doesn’t mean you can just forget about running containers and trust the applied security measures. Once the security measures are running, they can be attacked. We need to detect abnormal and unexpected behavior in our applications in order to take corrective action and prevent security incidents from happening again.

Many different attack vectors exist. For example MITRE ATT&CK provides an extensive list of tactics and techniques “based on real world observations”, which can be used both to apply prevention measures and to analyze the activity to detect abnormal behaviors, which can mean an attack or intrusion is being performed. The MITRE ATT&CK Matrix for Containers covers techniques specifically targeted against container technologies.

13. Set up real-time event and log auditing

Threats to container security can be detected by auditing different sources of logs and events, and analyzing abnormal activity. Sources of events include:

Host and Kubernetes logs
Cloud logs (CloudTrail in AWS, Activity Audit in GCP, etc.)
System calls in containers

Falco is capable of monitoring the executed system calls and generating alerts for suspicious activity. It includes a community-contributed library of rules, and you can create your own by using a simple syntax. Kubernetes audit log is also supported.

You can see nice examples of Falco in action in our Detecting MITRE ATT&CK articles:

Sysdig Secure extends the capabilities of Falco and can also ingest events from different cloud providers.

As an example, the following rule would trigger an alert whenever a new ECS Task is executed in the account:

rule: ECS Task Run or Started
condition: aws.eventSource="ecs.amazonaws.com" and (aws.eventName="RunTask" or aws.eventName="StartTask") and not aws.errorCode exists
output: A new task has been started in ECS (requesting user=%aws.user, requesting IP=%aws.sourceIP, AWS region=%aws.region, cluster=%jevt.value\[/requestParameters/cluster\], task definition=%aws.ecs.taskDefinition)
source: aws\_cloudtrail
description: Detect a new task is started in ECS.

Sysdig also includes an ever growing set of rules tagged with the corresponding compliance standards and controls, and provides a centralized dashboard for exploring all security events in your infrastructure:

14. Monitor your resources

Excessive resource usage (CPU, memory, network), quick decrease in available disk space, over-average error rate, or increased latency might be signals that something strange is happening in your system.

Collect metrics, like with Prometheus. Configure alerts to quickly get notified when the values exceed the expected thresholds. Use meaningful dashboards to explore the evolution of metrics, and correlate with changes in other metrics and events happening in your system.

In this example, we notice a sudden increase in request latency and the request rate falling. This could mean something is happening in your containers (e.g., a cryptominer consuming all the available CPU), or an exploit causing response slowness and potentially a DoS. Checking the related cluster events around that time frame, we see a pod has been replaced, so it is also possible that a malicious or simply incorrectly configured version was deployed.

Incident response and forensics

Once you detect a security incident is happening in your system, take action to stop the threat and limit any additional harm. Instead of just killing the container or shutting down a host, consider isolating it, pausing it, or taking a snapshot. A good forensics analysis will provide many clues and reveal what, when, and how it happened. It is critical to identify:

If the security incident was a real attack or just a component malfunction.
What exactly happened, where did it occur, and are any other potentially impacted components?
How can you prevent the security incident from happening again?

15. Isolate and investigate

When a security incident is detected, you should quickly stop it first to limit any further damage.

Stop and snapshot: When possible, isolate the host or container. Container runtimes’ offer was to “pause” the container (i.e., “docker pause” command) or take a snapshot and then stop it. For hosts, you might take a snapshot at the filesystem level, then shut it down. For EC2 or VM instances, you can also take a snapshot of the instance. Then, proceed to isolation. You can copy the snapshot to a safe sandbox environment, without networking, and resume the host or container.
Explore and forensics: Once isolated, you can ideally explore the live container or host, and investigate running processes. If the host or container is not alive, then you can just focus on the snapshot of the filesystem. Explore the logs and modified files. There are tools like, Sysdig Secure captures, that greatly enhance forensics capabilities by recording all the system calls around an event and allowing you to explore even after the container or host is dead.
Kill the compromised container and/or host as a last resource: Simply destroying the suspicious activity will prevent any additional harm in the short term. But missing details about what happened will make it impossible to prevent it from happening again, and you can end up in a never ending whack-a-mole situation, repeatedly waiting for the next attack to happen just to kill it again.

Check out a great example of forensics investigation in THREAT ALERT: Crypto miner attack involving RinBot’s server, a popular Discord bot.

16. Fix misconfigurations

Investigation should reveal what made the attack possible. Once you have discovered the attack source, take security measures to prevent it from happening again. The cause of a host, container, or application being compromised can be a bad configuration, like excessive permissions, exposed ports or services, or an exploited vulnerability.

In the case of the former, fix the misconfigurations to keep it from happening again. In the latter case, it might be possible to prevent a vulnerability from being exploited (or at least limit its scope) by making changes in configurations, like firewalls, using a more restrictive user, and protecting files or directories with additional permissions or ACLs, etc.

If the issue applies to other assets in your environment, apply the fix in all of them. It’s especially important to do so in those that might be exposed, like applications that are reachable from the internet if the exploit can be executed over a remote network connection.

17. Patch vulnerabilities

When possible, fix the vulnerability itself:

For operating system packages (dpkg, rpm, etc.): First check if the distribution vendor is offering an updated version of the package containing a fix. Just update the package or the container base image.
Older distribution versions: The vendor will stop providing updated versions and security fixes. Keep your hosts and images using supported versions before it is too late.
For language packages, like NodeJS, Go, Java, etc.: Check for updated versions of the dependencies. Search for minor updates or patch versions that simply fix security issues if you can’t spend additional time planning and testing for breaking changes that can happen on bigger version updates. But plan ahead: old versions won’t be maintained forever.
In case the distribution does not offer a patched version or there is no fix for unmaintained packages: It is still possible that a fix exists and can be manually applied or backported. This will require some additional work but it can be necessary for packages that are critical for your system and when there is no official fixed version yet. Check the vulnerability links in databases like NVD, vendor feeds and sources, public information in bug reports, etc. If a fix is available, you should be able to locate it.

If there is no fix available that you can apply on the impacted package, it might still be possible to prevent exploiting the vulnerability with configuration or protection measures (e.g., firewalls, isolation, etc.). Also, it might be complex and require a deep knowledge of the vulnerability, but you can add additional checks in your own code. For example, a vulnerability caused by an overflow in a JSON processing library that is used by a web API server could be prevented by adding some checks at the HTTP request level, blocking requests that contain strings that could potentially lead to the overflow.

18. Close the loop

Unfortunately, host and container security is not a one way trip where you just apply a set of security containers good practices once and can forget forever. Software and infrastructure are evolving everyday, and so complexity increases and new errors are introduced. This leads to vulnerabilities and configuration issues. New attacks and exploits are discovered continuously.

Start by including prevention and security best practices. Then, apply protection measures to your resources, mostly hosts and workloads, but also cloud services. Continue monitoring and detecting anomalous behavior to take action, respond, investigate and report the discovered incidents. Forensics evidence will close the loop: fix discovered vulnerabilities and improve protection to start over again, rebuilding your images, updating packages, reconfiguring your resources, and create incident reports to the future security incidents.

In the middle, you need to assess risk and manage vulnerabilities. The number of inputs to manage in a complex and big environment can be overwhelming, so classify and prioritize to focus on the highest risks first.

Conclusion

We’ve reviewed how container security best practices can be easily applied to your DevOps workflows. In particular, remember to:

Shift left security, the first step is prevention.
Protect all your assets.
Know everything that happens in your organization, monitoring and detecting issues as fast as possible.
Plan for incident response, because attacks are inevitable.

Remember that container security best practices don’t just include the delivered applications and container images themselves. You also need to include the full component stack used for building, distributing, and specifically executing the container.

54 percent of containers live for five minutes or less, which makes investigating anomalous behavior and breaches extremely challenging.

One of the key points of cloud-native security is addressing container security risks as soon as possible. Doing it later in the development life cycle slows down the pace of cloud adoption, while raising security and compliance risks.

Secure software supply chain: why every link matters

Miguel — Tue, 16 Nov 2021 10:23:06 +0000

The new threats in software development are not only related to the specific company itself. The whole software supply chain is a target for attackers and it is really important to make sure that we put all our effort into securing each link because if one fails, everything will be affected.

Supply chain activities include each step of the transformation of raw materials, components, and resources into a completed product, and its delivery to the end customer.

Each step could be a complex process itself and cause a security incident.

What is a software supply chain

The software supply chain is similar to other activities or industries. Some resources are consumed, then transformed, through a series of steps and processes, and finally supplied as a product or service to a customer.

In software, the raw materials are common libraries, code, hardware, and tools that transform code into a final deliverable. This deliverable can be deployed as either a user-facing application, a service (starting over with the same supply chain loop), or another package artifact that is included as a dependency, part of a different product.

The chain can be long and quite complex, so let’s try an extremely simplified example:

In order to produce the final “web application” for our customer, we need to transform (compile) a source code and consume information from third-party services. The source code itself depends on external libraries, which are produced from another code, etc.

Software supply chain attack happens when some malicious element is introduced in this chain.

A successful attack in any link of the supply can propagate the compromised code or component downstream, completely unnoticed, and cause mayhem across different stages.

In fact, many of these attacks focus on compromising a software vendor by injecting some malware or vulnerability at an intermediate step in order to exploit the final customers with fatal consequences.

As your company or product is just a piece in the overall software supply chain, security measures related to the supply chain can be applied on three different points:

Inputs: Library and package dependencies, third-party tools, software, services, or any artifact you are consuming, either public or from a private vendor.
Outputs: Guarantee the integrity of your deliverables and provide ways to verify the components downstream.
Processes and infrastructure: Protect your network, identities, credentials, signature keys, repositories, and processes.

We will explain some of the common supply chain attacks, with recent examples, and provide tips and practices that you can apply to mitigate the risks.

What is a supply chain attack

We already gave a definition of the Software supply chain with a simplified example, talking about code, libraries, and services. But let’s dig a bit deeper and put the focus on the source code:

Zooming in on any link on the chain will reveal additional details:

In the example, your source code will live in a private git repository, which could be part of your infrastructure or SaaS provided by a vendor, as well as compiler tools, base container image registries, etc.

Some dependencies are hosted in public repositories, like Docker Hub or Quay.io, and could be compromised. Also, we are publishing our application as a container image in a public repository too.

Some of the components in the chain (blue) are under your umbrella, like your private source code git repository, the application code itself, and the final binary or container image that is produced. But many other components or services (in green) are public services and resources, or provided by other companies, totally out of your control.

So a software supply chain attack can target you directly, or it can target any upstream element (like external dependencies or provided services), so you become a victim, either by directly suffering the attack, or by becoming a supplier of compromised resources.

Examples of supply chain attacks

Trends show that supply chain attacks are increasing at an exponential rate of 4-5x per year, with several thousand last year, the most common being related to dependency confusion or typosquatting, followed by malicious source code injection.

Fortunately, not every attack has a big enough impact to appear in the newspaper, but let’s analyze some of the most relevant and recent ones. Many other examples of different types of supply chain attacks are also collected by the CNCF in their Catalog of Supply Chain Compromises.

CodeCov

Leaked credentials in a Docker image from CodeCov allowed the attackers to modify a bash script. This script was downloaded and executed by customers, resulting in leaked customer credentials and attackers getting access to their git repositories.

Multiple customers like Monday.com, Hashicorp, Twilio, or Confluent were affected.

Solarwinds

The attackers infiltrated Solarwinds’ network and managed to inject malicious software into their build process. This malware (related to APT29, Nobelium) was bundled as part of Orion (a Network Management System) product updates. As part of the build process, the artifact was digitally signed and then downloaded by hundreds of customers.

Once the malware was running in the customer networks, the attackers spied and stole their information.

This incident also shows how a supply chain attack can propagate downstream and impact multiple customers: as part of the compromised Orion software, a TLS certificate private key for a mail server in Mimecast (a cloud cybersecurity services company) was leaked. This allowed attackers to perform a man-in-the-middle attack on the mail server and access customer emails.

We might never know the real impact of the Solarwinds attack, as the compromised software was running for several weeks or months inside networks of potentially thousands of customers.

And it is not yet finished: the attackers behind Solarwinds have been recently attempting to replicate similar attacks but to different parts of the supply chain, like resellers and other technology service providers that customize, deploy, and manage cloud services and other technologies on behalf of their customers.

Kaseya

It’s quite similar to the Solarwinds attack, but in this case, the attackers exploited a zero-day vulnerability in Kaseya systems. Once they took control of the systems, they executed remote commands in the customers’ systems using VSA, a remote management and monitoring tool.

Apple Xcode and XcodeGhost

In this attack, a trojanized version of the legitimate Xcode project “TabBarInteraction” was published in a public repository. Developers using the fake version of the project were inadvertently executing a script on each project build. The script opened a connection to a C2 (command and control) server.

We can find another example where repackaged versions of Xcode, with malicious code, were uploaded to Chinese file hosting services. Developers downloading the compromised version from these mirrors ended up with a modified object file. The object file is linked in the final executable when creating iOS applications. At least two well-known applications reached the official Apple AppStore, successfully passing Apple certification and code review.

NPM package ua-parser-js

On Oct. 22, 2021, developers of a very common NPM package, ua-parser-js, discovered that some attackers uploaded a compromised version of the package containing malware for Linux and Windows, and were capable of stealing data (at least passwords and cookies from the browser).

This ua-parser-js library is part of the supply chain for a large list of software and companies, including Facebook, for some of its user-facing applications in millions of computers.

Unicode and code compilers – Invisible vulnerabilities

Some researchers at the University of Cambridge claim the discovery of a novel supply-chain attack targeting source code in the subtleties of text-encoding standards, such as Unicode. The way Unicode works, it is possible to add special characters to the comments or some parts of the code that make the “logical encoding” of the code work in a different order from the way it is displayed. This makes it possible to hide malicious code inside code that looks correct to humans, bypassing review procedures.

But before panicking, please note the issue might not be that new and invisible, and it can be prevented by establishing trust in the code contributors and doing proper reviews.

The Huawei Ban

Finally, we include this example not as a known software supply chain attack, but as a demonstration of the importance of trusting providers in every single link of the chain. Even the hardware supply chain can be a target. For years, the U.S. government warned the world and American companies about security threats in Chinese network equipment:

Based on available classified and unclassified information, Huawei and ZTE cannot be trusted to be free of foreign state influence and thus pose a security threat to the United States and to our systems.

The warning seems to be based on the lack of information and the inability to satisfy some security concerns, and not on real evidence of any existing backdoor in the systems.

So we might never get real evidence, but the lack of confidence was so severe that Huawei was banned in May 2019 from doing business with any organization that operates in the United States.

Supply chain taxonomy (and controversy)

Not everybody agrees that an attack targeting any stage of the software supply chain falls into the category of “supply chain attack”. The European Union Agency for Cybersecurity (ENISA) in the Threat Landscape for Supply Chain Attacks report “proposes a taxonomy to characterize supply chain attacks and structure their subsequent analysis”:

But on the same report, they add:

“If no customer is attacked, or no supplier attacked, then it is probably not a supply chain attack.”

Explicitly, they mention some examples that don’t meet the requirements, like:

Some name-squatting or brand-hijacking packages were uploaded to public repositories because they did not compromise existing packages or the software repositories.
Discovered vulnerabilities or unintentional errors that were not used in known attacks, because they don’t target a supplier or a customer.
Public-facing vulnerable versions of software that allowed exfiltration of customer data, but in this case the supplier was not compromised (the attack was targeted as French IT providers) and the vulnerabilities were not intentional nor specifically crafted for this attack.

So for some incidents, there is not even a global consensus whether it can be classified as a supply chain attack or not.

Securing the software supply chain

Attacks to the software supply chain are broadening the attack surface of companies, as their security does not only depend on internal procedures. Now, they also need to ensure that third parties (including software, hardware, services, etc.) are not a gateway to attackers who can affect them.

Like traditional security, it is impossible to secure everything, especially as new kinds of software supply chain attacks are being discovered continuously.

Let’s explain where you need to focus on each of the layers to be as protected as possible.

Inputs of your software development

As a consumer of software artifacts, hardware, and services from multiple providers, ensure that your providers apply a high level of security.

The software supply chain is becoming so critical that in May 2021, the president of the United States issued an Executive Order targeted to improve the nation’s cybersecurity. As a result of the order, the NIST produced the Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 guide, which recommends minimum standards for verification by software vendors or developers.

Because it is impossible to achieve and guarantee 100 percent security, please also pay attention to any risks or incidents impacting your providers, and be ready to take quick corrective action in case a compromised component is either detected or reported.

Software development in your company

With regard to code and development processes, the mentioned Recommended Minimum Standards for Vendor or Developer Verification (Testing) of Software Under Executive Order (EO) 14028 includes the following set of techniques as the minimal safety requirements for software development life cycle. Make sure you don’t miss any of them:

Apply threat modeling to identify key or potentially overlooked testing targets.
Automated testing.
Code-based (static) analysis, using a code-scanner, and review for hard-coded secrets.
Dynamic analysis, with built-in checks and protections, black-box and fuzzy testing, web-app scanner, etc.
Apply similar checks to included software (third-party dependencies).
Fix critical bugs as soon as possible.

With Infrastructure as Code (IaC), threats in the software supply chain now potentially target not just software and applications, but the underlying infrastructure too.

The same software verification principles should be applied to the deployment and management of infrastructure as code, shifting left security. Sysdig’s acquisition of Apolicy was intended to improve our customer’s KSPM (Kubernetes Security Posture Management) and CSPM (Cloud Security Posture Management) use-cases by addressing IAC security in a unique and highly differentiated manner.

But security must be pervasive in all development stages and the whole company culture and practices. Securing the code and the development process is clearly not enough, as outlined by multiple organization researches and guides:

The Defending Against Software Supply Chain Attacks guide from Cybersecurity and Infrastructure Security Agency considers that the Software Supply Chain Lifecycle has six phases where “software is at risk of malicious or inadvertent introduction of vulnerabilities”:
- Design
- Development and production
- Distribution
- Acquisition and deployment
- Maintenance
- Disposal
The Cloud Native Computing Foundation (CNCF) keeps a living, community maintained document: the Software Supply Chain Security Paper. This paper aims to contribute to the community with “a series of recommended practices, tooling options, and design considerations that can reduce the likelihood and overall impact of a successful supply chain attack”.
The CNCF, Linux Foundation, VMware, Intel, Google, and others are also working on SLSA – Supply-chain Levels for Software Artifacts, a security framework, and a common language for increasing levels of software security and supply chain integrity for anyone working with the software. Each level provides an increasing degree of confidence, a way to say that software hasn’t been tampered with and can be securely traced back to its source.

Kubernetes and containers are so common nowadays that NSA/CISA also released a Kubernetes Hardening Guidance, highlighting “Supply chain risks” as one of three sources of compromises, and proposing the following hardening measures and mitigations:

Scan containers and Pods for vulnerabilities or misconfigurations.
Run containers and Pods with the least privileges possible.
Use network separation to control the amount of damage a compromise can cause.
Use firewalls to limit unneeded network connectivity and encryption to protect confidentiality.
Use strong authentication and authorization to limit user and administrator access, as well as to limit the attack surface.
Use log auditing so that administrators can monitor activity and be alerted to potential malicious activity. Our blog post, “Chain”ging the Game – how runtime makes your supply chain even more secure, provides nice examples of runtime threat detection.
Periodically review all Kubernetes settings and use vulnerability scans to help ensure risks are appropriately accounted for and security patches are applied.

Your software is the input to other companies or users

Everything you apply when choosing your best providers with a high level of confidence will apply to your company when acting as a supplier for other companies or for end-users. Even if you correctly implement security into every step and process of your software supply chain life cycle, you still need to manifest it and add specific measures to the delivered artifacts.

Provide evidence of regulatory compliance and security certifications within your organization that applies to the software supply chain.
Add a Software Bill of Materials (SBOM) as a way to track dependencies and third-party sources of compromise in your software.
Include digital signatures to prevent tampering and verify the source of your artifacts.
Use safe distribution channels, encrypted communications, and trusted hosting or storage infrastructure.

Conclusion

Software supply chain attacks are not really something recent. Ken Thompson, after receiving the Turing Award with Dennis Ritchie in 1984, wrote this speech about the importance of trust in the [code] providers. He shows how a trojanized version of the compiler binary produces modified versions of the “login” Unix command with a backdoor.

“To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software.”

But the ever-growing complexity of software, infrastructure, and dependencies, and the rate of growth of attacks targeting the supply chain, are making industries and organizations more and more concerned about its security.

At this point, you already noticed that the software supply chain is a very complex network of interconnected and heterogeneous pieces, from code and libraries to hardware components. So the approach for securing every single piece and link is very different and cannot be covered as a whole. But, whether you are a supplier or a consumer (or both), you need to put the focus on securing your processes and establishing strong connections with trusted and verified providers. Because even if you apply the best security practices and put all efforts in your own code, infrastructure, etc., you still depend on third-party components completely outside of your control. And the security of the software supply chain depends on every individual link.