DEV Community: Oshrat Nir

Yet another reason why the xz backdoor is a sneaky b@$tard

Oshrat Nir — Wed, 03 Apr 2024 12:57:43 +0000

This post was originally posted on the ARMO Security Blog and was authored by Ben Hirschberg CTO & Co-founder of ARMO

Background

If you just woke up from hibernation and you didn’t know about the back door that was implanted in some OpenSSH releases, this section is the TL;DR.

A contributor to the liblzma library (a compression library that is used by the OpenSSH project, among many others) submitted malicious code that included an obfuscated backdoor. Since the maintainers had no reason to suspect foul play, they accepted and merged the contribution. The malicious code made it into the compression library release, and later on to the OpenSSH server, which relies on the library in question. Any attacker with network access to the listening port of the OpenSSH server could run arbitrary processes on the server side and be able to communicate with this new process.

Problems

Now, there are a myriad of things that can be discussed here about supply chain attacks. It has been (justifiably) a hot topic in the Cybersecurity field to talk about supply chain security since the Solarwinds attacks. Talking about the importance of awareness around supply chain security is like talking about the awareness of social engineering threats. On the one hand, it’s a no-brainer that it is a problem. On the other hand, you cannot talk about it enough to keep people aware of it.

However, I want to talk about the XZ attack and backdoor from a different angle. The code that became part of OpenSSH was well planned, well executed, and well obfuscated by the attackers, making it even more sneaky than you would have thought.

Detection of an exploit at the network level

Let’s talk about the actual way that someone can detect an exploit of this backdoor.

There is a long line of runtime detection tools, both in the open-source and in commercial spaces. There are myriad intrusion detection tools (IDS) and web application firewalls (WAF) that are there to detect a malicious actor trying to exploit such a backdoor. In this specific case, the ability for such tools to be able to detect the exploitation of this backdoor at the network level, without specific knowledge about it, is near zero.

Why? Because of the way the backdoor is activated. It was completely embedded into the existing protocol so the payload looks like an extension to the SSH protocol. Any tool trying to detect an intrusion by only looking at the network layer, without knowing the specific mechanics of this backdoor, won’t be able to differentiate it from any other legitimate SSH packet. In other words, if this is a zero-day (meaning, no prior knowledge), there is no way that IDSs or Firewalls would catch it. If the vendor/operator knows what the activation packets look like, they can release the proper rules that will detect it.

Host level detection

Now let’s talk about host-based intrusion detection tools. EDRs, XDRs, and the like. A server process opening arbitrary sub-processes and connecting them to sockets is a red flag for any host-based runtime detection tool.

I would even dare to call this low-hanging fruit from the detection side. Without question, the attacker’s jackpot is to be able to open a process on the victim’s host and communicate with it. Therefore, detection tools have been trained and designed to detect arbitrary processes popping up and communicating over the network.

However, in this specific case, there is one big caveat and it is the very way that the SSH protocol, and as a result the server, were designed to work. SSH’s basic functionality is connecting a remote machine securely and running a terminal (shell). The underlying design of the SSH system is that you can run any executable remotely and running a shell is just a specific use-case of the general capability. Therefore, all server implementations can open arbitrary processes and connect to the network peer to communicate with them.

If you take this into account, you understand that there is something different here from the detection side, since the normal behavior of the server is to be connected over TCP and open arbitrary processes. Therefore, if you have a back door that behaves in the same way, i.e. opening an arbitrary process on the host, and connecting it to the TCP network, it is much harder to detect and differentiate malicious activity from normal behavior.

The most respected detection and monitoring tools, meant to detect an attack, look at the user space processes from the kernel side and see system calls made by the protected processes. In this very case, when an SSH server opens a new session, by design it creates a child process and connects the client TCP socket. What will happen in the case of this backdoor exploitation? The very same as the SSH process! It will fork and your process will invoke a new process image by calling into execv and connect it to through the stdout and stdin file descriptors to the client socket. It is the very same behavior of any normal user.

Due to this reason, I find this backdoor to be even more serious than initially presumed. Even the final line of defense, the runtime detection tools, would struggle to detect it

Falco example

As an example of the above point, let’s see how Falco behaves when it monitors an SSH server.

When Falco is running on a Kubernetes cluster where an SSH server is running in one of the Pods and a login is made, the following alert pops up:

05:15:41.910084560: Notice Redirect stdout/stdin to network connection (gparent=containerd-shim ggparent= gggparent= fd.sip=127.0.0.1 connection=127.0.0.1:47210->127.0.0.1:22 lport=47210 rport=22 fd_type=ipv4 fd_proto=fd.l4proto evt_type=dup2 user=root user_uid=0 user_loginuid=-1 process=sshd proc_exepath=/usr/sbin/sshd parent=sshd command=sshd -D terminal=0 container_id=1d065f4a3388 container_image=docker.io/rastasheep/ubuntu-sshd container_image_tag=18.04 container_name=ssh-server k8s_ns=default k8s_pod_name=ssh-server-856d78867c-jb9hf)

This is an alert that is expected. The Falco user is notified that in the ssh-server container, a stdin/stdout pipe was redirected to a network socket. This is a well-known attack technique, where an attacker tries to open a reverse shell on a remote machine.

However, in the case of an SSH server, this is its basic functionality. Therefore a user of an SSH server will either ignore these alerts from Falco or remove this rule to prevent noise.

In either case, an exploit of the backdoor would have gone unnoticed since it would’ve triggered the same alert(s).

Takeaways

What can be learned from this?

A successful supply chain attack does not end by injecting malicious code into the victim’s environment. An attacker, especially an advanced attacker, is also thinking about how the backdoor will be exploited.

The attackers’ intentions and thought processes are not known to us. I don’t know for sure if the command injection backdoor was implanted in OpenSSH deliberately to avoid runtime detectors. It is common to run SSH servers as root processes (not a good practice though). Maybe this was the reason. But as of the time of writing this blog, the consensus of the security community is that this attack was made by a state actor, and it is very possible that by design this attack was meant to circumvent detection tools.

Bottom line (literally) – it is crucial to invest more in the awareness of supply chain attacks in the open-source community and identify ways to prevent the next attack.

Leverage GitOps for Kubernetes security and Compliance

Oshrat Nir — Thu, 11 Jan 2024 11:03:00 +0000

Kuberenetes security and GitOps are 2 discrete disciplines. But, can they be better together?
You can find the answer to this in a recent blog post from ARMO.

Why context matters in Kubernetes security

Oshrat Nir — Wed, 10 Jan 2024 14:12:12 +0000

Background

There are things in the world that are absolute, and there are things that are relative. For example, it is an absolute truth that the middle three Star Wars episodes were better than the prequel three. But if we are talking about security, it is mostly accepted to be relative as it is a well-accepted thesis that there is no absolute security. Every system can eventually be broken. Like in the joke about the two folks who are attacked by a lion. One of them starts to run for his life, the other starts to tie his sport shoes. The first ask cynically, “what are you doing? Are you trying to outrun the lion?!” The second answers: “I don’t need to outrun the lion, it is enough if I outrun you!”.

Now that we have established that there is no absolute security, we have to take the context into account in order to understand the security posture of a system. And no, I don’t mean to check on who else is running from a lion! To tell if a vulnerability, or a suboptimal setting can be exploited to breach a system, you need to have a full picture of how the system is put together, and how it’s actually being used. If not, you may find yourself fixing security issues that don’t really matter. Thus wasting two of your most important resources – time and money.

Kubernetes security is inherently complex due to the intricate and dynamic nature of the modern cloud-native environments. As an orchestration platform, Kubernetes manages numerous containers, each running different applications, often across various nodes and logical groupings (namespaces). This decentralized architecture, while offering scalability and flexibility, introduces multiple layers of potential security risks – from the application level down to the infrastructure. Moreover, the constant evolution of cloud-native technologies means that Kubernetes environments are continuously exposed to new security challenges. These challenges are compounded by the need for stringent access controls, network policies, and the secure management of sensitive data like secrets and configuration files. Effective Kubernetes security requires a comprehensive approach that encompasses application, infrastructure, networking, access control and an understanding of the constantly evolving threat landscape.

Kubernetes’ many security features may be deceiving, giving the impression that they cover all bases. While they address most problems related to platform security and offer some extensions around applications and networks (such as Kubernetes Secrets or Native Network Policies), these do not automatically translate to securing everything layered on top or below. Many of these are already included in existing fields such as “AppSec,” “Platform security,” and “Cloud security.” ARMO believes that using their (apps, platform, and cloud) information streams to provide context to Kubernetes security yields the best results. It provides users with a better understanding of the system’s overall security posture.

To put this in perspective, let’s use an example. Think about an application that has a vulnerability that can be exploited by an attacker who has network-level access to the application. If we know that this application can only receive requests from other applications from inside a cluster, that’s one thing. But if it can be accessed from the public internet, then the problem is in a completely different ballpark.

Information streams

In a Kubernetes cluster, security practitioners have access to plenty of information streams that are important for ensuring the security and integrity of the environment. Here’s a breakdown of these key streams:

Kubernetes API

The Kubernetes API is the central nervous system of a Kubernetes cluster, offering comprehensive insight into the state and management of cluster resources. Security practitioners can monitor the state of the cluster using the API, making sure that the configuration is secure and detect unusual patterns in the state of the cluster. This information is also invaluable to put other information streams into context since people are not working with classical constructs like “TCP connections” or “processes” alone but in the context of Kubernetes workloads.

Audit Logs

Kubernetes audit logs are invaluable for security monitoring and forensic analysis. They record a chronological account of requests made to the Kubernetes API, detailing who made the request, what was requested, and the outcome of the request. This data is crucial for detecting and investigating security incidents, ensuring compliance, and understanding the context of operational changes.

RBAC Settings

Role-Based Access Control (RBAC) settings in Kubernetes are fundamental for enforcing the principle of least privilege. By examining RBAC configurations, security practitioners can ensure that users and services have only the permissions necessary for their function, minimizing the risk of privilege escalation and unauthorized access.

Source: ARMO Platform

Image Vulnerability Information

Container images used in Kubernetes typically contain vulnerabilities. Security teams must stay informed about any vulnerabilities within these images to take timely action. Tools that scan container images for known vulnerabilities and provide detailed reports are essential in maintaining a secure container environment.

Node Information

The security of a Kubernetes cluster is heavily dependent on the security of its nodes. This includes understanding the Operating System (OS) details, ensuring they are up-to-date with patches, and validating installation configurations. Monitoring node health and security posture is critical to safeguard the underlying infrastructure that supports the cluster.

eBPF Application Observability Information

eBPF (extended Berkeley Packet Filter) enables detailed observability and security monitoring within a Kubernetes cluster. It provides deep visibility into processes, file activity, network traffic and system calls at the kernel level without modifying the kernel or the applications. This allows for real-time security monitoring and performance troubleshooting, offering security practitioners a powerful tool to observe and secure applications running in the cluster.

By leveraging these information streams, security practitioners in a Kubernetes environment can achieve a comprehensive understanding of the cluster’s security posture or detect security incidents. This information helps pinpoint and quickly counter security risks, keeping the cluster strong against ever-changing threats.

Examples of leveraging multiple streams for use cases

The idea is that cybersecurity subject matter experts, working within recognized organizations create security guidelines that give practitioners a solid foundation. However, when applying them to a variety of business scenarios, they are often found to be too generic. As a result, security professionals must modify the general rules to fit their specific infrastructure and unique application portfolios. By enhancing static security with the data streams mentioned above, ARMO Platform is automating this enrichment process and supporting the following security use cases.

CVE reachability

CVE Reachability is a new approach in container security, leveraging the power of eBPF to enhance the accuracy and efficiency of vulnerability management. This technology focuses on the utilization of eBPF to track file access information within a workload during container runtime. Here’s how it fundamentally changes the security landscape:

At the core of CVE Reachability is the understanding that not all software packages in a container image are actively used when the container is running. Traditional vulnerability scans of container images often flag all known vulnerabilities in every package included in the image, regardless of whether those packages are in use. This approach often results in a large number of false positives, burdening security teams with the task of sifting through these results to identify genuine threats.

By employing eBPF, CVE Reachability discerns which software packages are actually used during container runtime. It monitors and records the file accesses made by the container, providing precise data on which parts of the container image are active. With this information, security practitioners can then filter the results of vulnerability scans to focus solely on the vulnerabilities present in the active packages.

Source: ARMO Platform

This targeted approach has several significant benefits:

Reduction of False Positives – By eliminating dormant packages from vulnerability scan results, CVE Reachability can reduce false positives by up to 80%. This guides the security teams to focus on vulnerabilities that pose a real threat..

Enhanced Security Posture – With a more accurate understanding of the active attack surface, organizations can prioritize patching and mitigation efforts more effectively. This leads to a stronger and more resilient security posture.

Operational Efficiency – Security practitioners can allocate their time and resources more efficiently, as they no longer need to investigate vulnerabilities in unused packages. This not only saves time but also reduces the complexity involved in securing containerized applications.

In conclusion , CVE Reachability represents a significant advancement in container security. By leveraging eBPF for real-time analysis of container workloads, it offers a more precise and efficient method for managing vulnerabilities, significantly reducing the burden on security teams and enhancing the overall security of Kubernetes environments.

Hardening cloud workloads

Implementing the least privilege principle for Kubernetes-based workloads is a challenging yet important aspect of securing containerized environments. The principle of least privilege requires that applications and processes operate with the minimum levels of permissions necessary to perform their functions. Achieving this in Kubernetes, just as in other complex systems, is not straightforward due to its complexity and the variability of application behaviors. To effectively tighten an application’s privileges, it’s crucial to have a deep understanding of its behavior. This includes knowing which resources it accesses, what operations it performs, and how it interacts with the system and network. Without this knowledge, defining the minimum necessary privileges is largely a guesswork. Here are some examples of how eBPF can be instrumental in hardening workload configurations.

Let’s take for example Linux capabilities. eBPF can be used to monitor a workload’s runtime behavior, specifically tracking the Linux capabilities it uses. Linux capabilities are a fine-grained access control mechanism that divide the privileges traditionally associated with superuser access into distinct units. By observing which capabilities a workload actually utilizes, security teams can tailor the permissions to fit the workload’s specific needs, ensuring that only necessary capabilities are enabled.

Limiting System Calls with Seccomp Profiles

System calls are how a program requests a service from the operating system’s kernel. eBPF can monitor the system calls made by a workload, providing exact insight into its interaction with the OS. With this data, security teams can create seccomp (Linux Secure Computing Mode) profiles that precisely define which system calls are permitted for a given workload. This level of control is typically hard to achieve manually due to the vast number of potential system calls and the difficulty in predicting which ones an application will require.

Source: ARMO Platform

Network Policies

eBPF’s observability capabilities extend to network interactions as well. By monitoring which external entities a workload communicates with, eBPF can help in creating accurate network policies. These policies can restrict network access to only those communications that are necessary for the workload’s operation, thereby reducing the attack surface and preventing potential lateral movement of threats within the network.

Source: ARMO Platform

Detecting attack paths

The concept of attack paths in a Kubernetes environment revolves around the identification and analysis of potential routes an attacker could potentially exploit to compromise the system or its assets. Detecting these attack paths requires a multi-dimensional approach, considering various aspects of the cluster’s configuration and state. To effectively identify potential attack paths, it’s necessary to integrate and analyze information from multiple sources within the Kubernetes ecosystem. This includes data from network configurations, workload behaviors, security vulnerabilities, access controls, and more.

Consider a Kubernetes workload that is configured to receive traffic from the public internet. This configuration detail can be gleaned from the Kubernetes settings, such as ingress rules or service types.

Suppose this same workload is identified by a vulnerability scanner as having critical security vulnerabilities. This information adds another layer to the security assessment. Combining this information results in flagging a heightened risk for this particular workload.

Furthermore, let’s say the workload has a mounted secret, which could be sensitive data or credentials necessary for its operation. This aspect introduces the potential for data exposure or unauthorized access if the workload is compromised.

By looking at this information collectively – the public internet exposure, the identified vulnerabilities, and the access to sensitive data – a comprehensive security system can deduce that the mounted secret is at risk of being compromised by external threats. This realization forms the basis of an attack path: an external actor could potentially exploit the vulnerabilities to gain unauthorized access to the secret.

This method of combining multiple data sources to identify attack paths can be applied to various other scenarios in a Kubernetes environment. Each scenario will have its own unique configurations, vulnerabilities, and access controls, which collectively indicate potential attack vectors.

Source: ARMO Platform

This approach underscores the importance of holistic security analysis in Kubernetes. By integrating data from multiple sources, security systems can map out possible attack paths Thus, enabling security teams to preemptively address vulnerabilities, tighten configurations, and safeguard sensitive data. Such proactive measures are vital in mitigating the risk of attacks and maintaining the integrity and security of Kubernetes clusters.

Conclusion

Looking at Kubernetes through various information sources, such as the Kubernetes API and audit logs, provides an understanding of the system’s security posture in context. By leveraging technologies like CVE Reachability and eBPF, container security can be enhanced. This leads to more accurate vulnerability management and workload configuration hardening. Integrating data from multiple sources allows security systems to identify potential attack paths and take proactive measures to address vulnerabilities. A holistic security analysis is crucial for maintaining the integrity and security of Kubernetes clusters. If you’re seeking to improve your Kubernetes security, try ARMO Platform today.

The post Why context matters in Kubernetes security appeared first on ARMO.

Kubernetes 1.29: The Security Perspective

Oshrat Nir — Sun, 03 Dec 2023 13:41:12 +0000

The Kubernetes 1.29 drop has been delayed by a few days to allow more tests with the latest Go 1.21 bugfix.

However, the team at ARMO reviewed the issues and the documentation and prepared a blog post, to give you a preview of what is coming up, through the lens of Kubernetes security.

Feel free to share your thoughts in the comments.

Kubernetes 1.29: The Security Perspective

Oshrat Nir — Sun, 03 Dec 2023 13:40:20 +0000

The Kubernetes 1.29 drop has been delayed by a few days to allow more tests with the latest Go 1.21 bugfix.

However, the team at ARMO reviewed the issues and the documentation and prepared a blog post, to give you a preview of what is coming up, through the lens of Kubernetes security.

Feel free to share your thoughts in the comments.

The 🐞 hunt starts today!

Oshrat Nir — Mon, 20 Nov 2023 07:20:22 +0000

Kubescape is joining up CodiumAI for a week-long bug bounty.

🎃 If you are still warmed up from hashtag hacktoberfest, that's great! Join and show us what you've got.

🤖 If not (and even if so) we encourage everybody to use an AI copilot, like CodiumAI, hashtag chatgpt or any other you like.

🎁 There will be prizes!

For more details: https://lnkd.in/eFZmsuYx

Kubescape 3.0 is available to enhance your K8s security experience

Oshrat Nir — Tue, 07 Nov 2023 17:36:58 +0000

Kubescape is the first Kubernetes security scanner that was accepted to the cloud native computing foundation. Kubescape 3.0 is a major release that extended the functionality of the original misconfiguration scanner to include vulnerabilities and usabillty improvements.
If you are interested in an overview, feel free to check out the blog post.
To dive straight into the code check out the repo.
I'd love it hear what you think. What you like, what can be improved and of course, if you have any questions, hit me up.

📣 Feature Announcement 📣 Advanced Side-by-Side remediation of Kubernetes misconfiguratuions 🔥

Oshrat Nir — Thu, 02 Nov 2023 10:49:57 +0000

Are you worried about misconfigurations in your Kubernetes environment? You're not alone. ARMO research shows that 100% of Kubernetes clusters tested have at least one misconfiguration, with 50% having over 14 of them. These misconfigurations can lead to serious security issues and need to be taken seriously.

That's why ARMO continues to leverage the #Kubescape misconfiguration scanner to provide a powerful solution for identification and remediation. And now, we are happy to announce the introduction of advanced side-by-side remediation. With this feature, you can remediate misconfigurations, with source and target displayed side-by-side, just like your favorite code review and maintenance interface on Github.

Not only do users get cues for in-place remediation, but the advanced side-by-side remediation feature takes the entire experience to a whole new level, making it simpler and more intuitive than ever before. Available from Helm version v.1.16.0 onwards, this feature is designed to empower developers of all skill levels in their journey towards secure and robust Kubernetes deployments.

Curious about how side-by-side remediation works and its benefits? Check out the introductry blog post. And if you're ready to see the impact on your quality of life, give it a spin today and take your Kubernetes security to the next level.

Join us in the fight against misconfigurations and keep your Kubernetes installations secure with advanced side-by-side remediation.

Feature announcement : Attack Path

Oshrat Nir — Mon, 30 Oct 2023 13:19:48 +0000

Whether you call it Attack Path or Attack Chain it is how an attacker accesses and moves about in your Kubernetes installation.

Introducing a graphic map to show you how an attack on your installation can unfold. It also shows you the weak points and provides advice on what to patch or how to remeidate.

Learn more here

Kubernetes 1.28 a security perspective

Oshrat Nir — Wed, 16 Aug 2023 05:54:34 +0000

Guess what happened yesterday?

Kubernetes 1.28 dropped and so did the ARMO's matching blog post. In the post we rounded up security features related to this version and gave a tl;dr for them.

So, as you dive into version 1.28, just make sure you keep #security in mind too.