DEV Community: Deepfence

Network Traffic Observability: Three PacketStreamer Use Cases

Deepfence — Mon, 09 May 2022 18:24:15 +0000

A few weeks ago, we announced a new open source project called PacketStreamer, which provides a simple, lightweight, scalable technique for capturing and streaming packets from virtualized environments (K8s, VMs, AWS Fargate) and across multiple clouds. PacketStreamer has generated lots of buzz and garnered widespread support from the community. Today, we’d like to dive a bit deeper into some of the specific ways to use PacketStreamer.

Security Use Case #1: Threat Hunting

The biggest driver for starting the PacketStreamer open source project was the desire to make threat hunting easier, especially for cloud native workloads. One of the best ways to identify threats is by observing network traffic, as it can reveal suspicious behaviors early on.

Modern compute environments are very different from the single Unix server assumptions that defined the design of legacy packet capture tools. Modern environments are cloud-based or distributed across many servers, meaning that packet capture must support centralized capture across distributed and elastic endpoints. They also may use virtualization technologies that limit access to the host and kernel, meaning that packet capture must be deployed via containers and DaemonSets.

PacketStreamer is the network capture solution for modern, cloud native environments. It captures traffic from large numbers of remote servers (for example, cloud nodes) and streams it to a centralized location. It supports modern stacks, such as Kubernetes (via a DaemonSet), Docker, and AWS Fargate, as well as standard hosts.

PacketStreamer creates efficiency by livestreaming the network traffic that an application receives to a different server – either on the cloud or a different system altogether. Essentially, it acts as a mirror, providing an exact replica of the traffic going into the application. From there, users can perform live analysis with their tool of choice. PacketStreamer is not an inline tool and therefore does not add any latency to the application while simultaneously enabling continuous monitoring of all the network traffic that comes to that application for any bad actors or malicious activity.

PacketStreamer can be used with any traffic analyzer and it can mirror as much traffic as the system on which it operates can handle, so it’s a great option for many organizations.

Security Use Case #2: Forensics

Security teams typically engage in post-facto traffic analysis following a security event. Traditionally, this means saving the traffic locally on the system where the application is running, extracting the file out of that system, and analyzing it at a later time. PacketStreamer simplifies this multi-step process by remotely saving the network traffic, thereby eliminating the additional steps of saving it locally, extracting the file, checking disc space, etc.

Users have the ability to remotely save the file to the location of their choice, and new remote save locations are added regularly. For example, we are particularly grateful that a member of the PacketStreamer community contributed to the project by adding a way to save to an Amazon S3 bucket.

Bonus Use Case: Debugging

While we built PacketStreamer to help the community enhance their security posture, we’re delighted that community members have found even more ways to use it. For example, we’ve heard from users that they find it helpful for debugging intermittent issues. It makes sense that PacketStreamer can help surface errors when log files aren’t detailed enough. By gathering network traffic in PacketStreamer, users can see the requests processed by a server to help pinpoint the root cause.

Whether you use PacketStreamer for threat hunting, forensics, or another use case altogether, we want to hear about it. Join us on Slack to ask questions and get answers, or just follow along with the conversation.

If you haven’t yet tried PacketStreamer, you can find it on GitHub.

Introducing PacketStreamer: Distributed Packet Capture for Cloud Native Platforms

Deepfence — Tue, 12 Apr 2022 17:00:36 +0000

PacketStreamer is an open source tool that captures network traffic from multiple remote sources concurrently and aggregates the data into a single pcap log file. It is written in golang and supports network capture from Kubernetes nodes, Docker hosts, and bare-metal/virtual-machine servers.

You can build and install PacketStreamer from GitHub: https://github.com/deepfence/PacketStreamer

Why Network Capture?

One foundation of a good cybersecurity practice is the ability to capture attack actor TTPs (Tactics, Techniques, and Procedures) from across and behind the attack surface. Tools such as Sysdig Falco capture TTP signals from running workloads (process changes, filesystem access, etc.), and can give indications of local compromise, but these signals alone only tell the late-stage story of an attack event.

Organizations need to see a bigger context, and that’s where network capture and analysis comes into play. Observing network traffic can reveal attacker behaviors before a successful compromise, such as reconnaissance activity and weaponization that is targeted at specific vulnerabilities. Observing traffic can also reveal lateral spread and exfiltration activities.

For example, in a log4j exploit, almost all of the initial signals are network-based. The initial JNDI recon against multiple workloads, the JNDI request that then triggers an outgoing request (beacon) to an attacker’s listener, the subsequent request that retrieves the Java class to be run… all of these are network activities and cannot be identified by on-workload sensors. The first signal you get from on-workload telemetry may be the installation of an exploit kit (a crypto-miner for example).

What Can I Do with PacketStreamer?

With PacketStreamer, you can extend your traffic capture activities to span large numbers of target systems. For example, if you run honeypot servers to gather attack TTPs, you can use PacketStreamer to listen for traffic and aggregate all captured traffic on a central receiver.

In the following example, we install PacketStreamer on three honeypot servers: a host with a basic WordPress installation, one with an inviting NGINX configuration that responds to every request with a 200 OK message, and a host running the honeydb service.

Honeypot Server

Our honeypot servers run a range of web and other services, and routinely receive recon traffic from remote hosts. We’ll use packetstreamer to capture the traffic and forward it to the target receiver:

# update sensor-remote.yaml to send traffic to the target
# receiver IP address and port
sudo packetstreamer sensor \ --config contrib/config/sensor-remote.yaml

Receiver Server

Our receiver server is located behind the firewall, listening on port 8081 for traffic from the honeypot sensors:

packetstreamer receiver --config contrib/config/receiver.yaml

The receiver server writes the aggregated capture traffic to a log file, such as /tmp/dump_file. You can watch and process that log file in a variety of ways, such as using tshark to decode selected protocols:

tail -c +1 -f /tmp/dump_file | tshark -r — -Y http 10388 66.320435 178.62.5.62 → 94.200.83.110 HTTP 312 HTTP/1.1 200 OK 10389 66.489650 94.200.83.110 → 178.62.5.62 HTTP 125 POST /wr54jj HTTP/1.1 11905 794.572402 86.171.162.177 → 46.101.77.119 HTTP 416 GET /wp-includes/js/wp-emoji-release.min.js?ver=5.8.4 HTTP/1.1 11907 794.573117 86.171.162.177 → 46.101.77.119 HTTP 441 GET /wp-includes/css/dist/block-library/style.min.css?ver=5.8.4 HTTP/1.1 11909 794.573576 86.171.162.177 → 46.101.77.119 HTTP 408 GET /wp-includes/js/wp-embed.min.js?ver=5.8.4 HTTP/1.1 12558 1204.781243 109.237.103.9 → 178.62.5.62 HTTP 295 GET /.env HTTP/1.1 12580 1205.040161 109.237.103.9 → 178.62.5.62 HTTP 307 GET /.aws/credentials HTTP/1.1 12593 1205.194548 109.237.103.9 → 178.62.5.62 HTTP 86 POST /.aws/credentials HTTP/1.1 (application/x-www-form-urlencoded) 13393 1352.414459 92.53.64.29 → 46.101.77.119 HTTP 599 POST /boaform/admin/formLogin HTTP/1.1 (application/x-www-form-urlencoded)Continuation 19020 3475.869367 91.98.167.220 → 178.62.5.62 HTTP 522 POST /tvm5f7 HTTP/1.1 Continuation 19027 3476.218166 91.98.167.220 → 178.62.5.62 HTTP 522 POST /ep6j56 HTTP/1.1 19042 3478.949728 91.98.167.220 → 178.62.5.62 HTTP 518 POST /y4p8vy HTTP/1.1 22661 4197.353628 2.50.89.16 → 178.62.5.62 HTTP 517 POST /q7e8vf HTTP/1.1 Continuation 22676 4198.930334 2.50.89.16 → 178.62.5.62 HTTP 520 POST /devret HTTP/1.1 24057 4763.594258 92.53.64.29 → 178.62.5.62 HTTP 593 POST /boaform/admin/formLogin HTTP/1.1 (application/x-www-form-urlencoded)Continuation

How Does Deepfence Use PacketStreamer?

PacketStreamer is also an integral part of the Deepfence ThreatStryker product. ThreatStryker gathers attack actor TTPs from cloud workloads and from network traffic. It classifies them to determine the TTP type and potential intent, and correlates the signals to determine how an attack is unfolding in real time.

To the best of our knowledge, there is no other simple, lightweight, scalable method to capture and stream packets from virtualized environments (K8s, VMs, AWS Fargate) and across multiple clouds. We’re open sourcing this tool to enable users to:

Capture and retain traffic for post facto analysis and forensics
Support threat-hunting activities across a broad target infrastructure
Experiment with new approaches such as ML against network traffic to detect anomalies

We’d welcome any feedback, contributions and suggestions. Please start with the PacketStreamer GitHub repository, and feel welcome to join the Deepfence Community Slack.

ThreatMapper 1.3.0: Now with Secret Scanning, Runtime SBOMs, and More

Deepfence — Tue, 15 Mar 2022 19:25:09 +0000

We are pleased to announce the general availability of ThreatMapper 1.3.0! Highlights from this latest release include:

Secret scanning at runtime
Runtime Software Bill of Materials (runtime SBOM)
A new approach to vulnerability scanning
Enhanced Attack Path Visualizations

Continue reading to learn all about the new features and how to get started.

Key Features in TheatMapper 1.3.0

Scan for Secrets in Production

In this release, we’ve integrated the open source project called SecretScanner into ThreatMapper so that now you can scan for both vulnerabilities and secrets in production, assess the risks associated across all potential issues, and then prioritize remediation accordingly.

With ThreatMapper, you can scan filesystems and container images looking for over 140 different secret types, including unprotected keys, tokens, and passwords. With this new capability, you will get a complete list of all sensitive secrets in your production environment, including those missed by traditional shift left scans performed during development. This is important because pre-production scans aren’t able to find secrets in host operating systems and third-party containers powering your applications in production, or legacy workloads created before your shift left measures were in place:

ThreatMapper 1.3 can identify more than 140 different secrets types

The integration of SecretScanner into ThreatMapper provides better defense against sophisticated multi-stage attacks. After an initial exploit, attackers seek ways to spread laterally and gain control of more hosts and workloads before reaching their final target. Sensitive secrets, like encryption keys, authentication tokens, or passwords, offer attackers ways to move laterally once they’re in.

While no one purposely hands over the proverbial keys to the castle, secrets sometimes make their way into production. Developers might use temporary secrets to generate build processes, but then forget to delete them after the image is built. Or, they might unknowingly apply weak security policies that embed overly permissive authentication tokens. It’s also common for workloads to acquire secrets dynamically when they are in production. Regardless of how they got there, it’s essential to find and fix sensitive secrets that make it into your production environments.

A full explanation of each result allows you to determine if it is a false positive (as in this case), or an issue that requires remediation

Generate a Runtime SBOM

Maintaining SBOMs for running applications and infrastructure is key to securing the software supply chain through increased transparency and information sharing. Standards and tooling are emerging that can process SBOM data for a variety of purposes. However, as we well know, not all runtime assets go through a formal CI process, so SBOM coverage is far from complete.

ThreatMapper 1.3.0 calculates runtime SBOMs for scanned workloads and makes these available for inspection through the UI and API. The runtime SBOM enumerates all of the packages and software items deployed in the workload, which may drift from the at-build-time SBOM. This new feature helps eliminate potential blindspots by identifying what goes into production and what changes during production. By tracking components and applying runtime context, the new runtime SBOM capability provides you with:

Smaller and more actionable asset inventories
More clearly defined attack surfaces to protect
Another way to detect threats based on deviations from pre-production SBOMs

The runtime SBOM can be inspected or downloaded as a JSON document

We’ll continue to build out the new runtime SBOM capabilities, working with emerging standards and tools and enhancing the static SBOMs with additional information that can only be obtained at runtime.

Experience New Vulnerability Scanning

Part of what enables ThreatMapper to generate runtime SBOMs is linked to a modification in how it scans running operating systems, applications, containers, and serverless workloads for vulnerable software components. In this release, we incorporated two new open source tools– Syft and Grype – into ThreatMapper. Although these are backend changes, they will open up new benefits for users, including:

Support for additional host operating systems
Faster initialization times, smaller data transfers from ThreatMapper’s lightweight sensors to the management console, and faster scanning to improve overall performance, particularly for large deployments
Integrations with many CI tools and two user communities

See Enhanced Attack Path Visualizations

We’ve extended our signature Attack Path Visualizations in this release. ThreatMapper now provides you with additional context about the most vulnerable attack paths in your environment by specifying each path’s exposure to the internet.

By categorizing paths with direct and indirect internet exposure, ThreatMapper helps you narrow down from thousands of potential issues to a handful that need fixing immediately.

See attack paths with direct and indirect internet exposure

This enhancement provides visual context about the depth of visibility that ThreatMapper offers. It exposes the easier-to-find attack paths with direct internet exposure along with the hard to find paths further downstream, hidden behind proxies and exposed to potentially malicious traffic indirectly.

Get the Latest!

We hope you’re as excited as we are about all the latest updates and improvements to ThreatMapper! It is our intent that ThreatMapper becomes an open platform on which users and partners continue to build integrations and solutions. Check it out and let us know what you think.

To install or upgrade to ThreatMapper 1.3.0, please refer to the detailed installation instructions.
Share your feedback by joining us in the Deepfence Community Slack.
Learn more in the upcoming webinar on March 29 at 10:00 AM PT.

Finally, a huge thank you goes out to our amazing community. Your ongoing feedback and contributions help us continue to shape ThreatMapper and unlock even more use cases.

...

Deepfence is dedicated to helping organizations secure their infrastructure and applications across the cloud native continuum. ThreatMapper open source scans, maps, and ranks vulnerabilities in running containers, images, hosts, and repositories. ThreatStryker elevates these capabilities by providing runtime attack analysis, threat assessment, and targeted protection.

Interested in learning more? Schedule a consultation with one of our security experts today.

Open Source Runtime Exploitability Management with ThreatMapper

Deepfence — Fri, 03 Dec 2021 21:03:39 +0000

CNCF Webinar - Move from mere vulnerability management to exploitability management with ThreatMapper

https://www.youtube.com/watch?v=S-sNUuLDhz0

ThreatMapper - Open Source Runtime Threat Scanner

Deepfence — Mon, 25 Oct 2021 19:20:47 +0000

We’re excited to announce today that ThreatMapper is now 100% open source under the Apache 2.0 license! If you’re not familiar with ThreatMapper, it’s a rapidly evolving cloud native security observability platform that scans, maps, and ranks vulnerabilities from development through production across serverless, Kubernetes, container, and multi-cloud environments.

Because modern applications and services rely heavily on shared, open source components, securing them is best done as a collaborative, community effort. By open sourcing ThreatMapper, we aim to help developers, DevOps, DevSecOps, and security teams identify and prioritize threats quickly and easily, and focus their efforts on the vulnerabilities that need to be fixed first.

When we initially launched ThreatMapper, we first made it available as a freemium edition and worked closely with dozens of early adopters to evolve it into the robust cloud native security platform that it is today. By working alongside security professionals securing modern application environments, not only were we able to build out a rich set of features and capabilities that solve real-world challenges, but we were also able to see ThreatMapper make a tangible impact on security teams.

Benefits of Using ThreatMapper

Here are just some of the benefits you get by using ThreatMapper to secure your applications and infrastructure:

See the topology of your applications and infrastructure: ThreatMapper auto-discovers your production infrastructure – including cloud instances, Kubernetes nodes, serverless resources, and containers – and maps the topology of your applications in real time.

Discover vulnerabilities, including fresh vulnerabilities in production that were not known at build or deploy time: ThreatMapper scans hosts, containers, and applications for known vulnerable dependencies, taking threat feeds from more than 50 different sources. ThreatMapper augments any “shift left” vulnerability scanning you may do in your development pipeline, and scans third-party components such as monitoring and load-balancing tools.
Rank vulnerabilities by attack surface: ThreatMapper ranks the discovered vulnerabilities based on CVSS and other severity signals, as well as their exploit method and proximity to your external attack surface. With ThreatMapper, you know what vulnerabilities pose the greatest threats, and what you must fix first.

Roadmap

ThreatMapper is a fully open source platform that makes it easy to scan for vulnerabilities and build a map of threats across multiple clouds and application types. ThreatStryker (our commercial offering) extends ThreatMapper with compliance scanning, runtime sensors, and a correlation and protection engine.

Our intent is to migrate all security and observability capabilities, including compliance scanning and runtime sensors, into the open source ThreatMapper platform. ThreatMapper will make all threat and runtime data available through public APIs, for dashboards, SIEM and other external applications to consume.

Summary & What’s Next?

Thank you to everyone who helped us on our journey so far to make ThreatMapper the robust open source security tool that we’re announcing today. We’re so grateful for the many design partners, customers, security professionals, advisors, and members of the Deepfence team (the Deepforce!) who helped us achieve this amazing milestone.

While open sourcing ThreatMapper was always on our roadmap — today is only the beginning! We’ll continue to build and release new features on our mission to protect the cloud native continuum.

Download, try, and contribute to open source ThreatMapper.