DEV Community: Px Mx

Anyone Can Have a Bad Day

Px Mx — Wed, 10 Jul 2019 15:21:16 +0000

Anyone can have a bad day. Take for example, in professional baseball, the league’s top-performing pitcher. This pitcher could be unhittable for most of the season, but then have a single game where he just can’t find the strike zone or the opposing team is hitting and scoring with ease. Yesterday, the CDN provider Cloudflare, had a bad day as the experienced a service outage resulting in about 30 minutes of downtime for customers. In baseball, the root of the problem for a bad day is mental, which is not easily avoidable. However, in technology, we stand a better chance of avoiding a bad day with testing and with the technology we choose to implement. In this post, I highlight the root cause of Cloudflare’s outage, but moreover, I discuss the risks of the cloud choices we have.

The Outage

As described in Cloudflare’s blog, there was a global software deploy that resulted in a CPU spike, or as they put it, “massive spike in CPU utilization”. This overconsumption of CPU handicapped their ability to process requests, effectively a denial of service (DoS). As a result, all users trying to access sites fronted by Cloudflare received a 502 Bad Gateway error response.

The Cause

The context of the outage causing software deploy was the Cloudflare Web Application Firewall (WAF). Specifically, it was a particular WAF rule, as an update to the blog states, “The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules.” The rule contained an erroneous regular expression that caused the CPU spike. Regular expressions are a sequence of characters that define a pattern for searching text. Regular expressions are traditionally used in WAF technologies to search application inputs to find patterns that match the signature of an attack payload. Regular expressions can be very complex, which of course requires CPU to process. The more complex, the more CPU required. One bespoke risk with complex regular expressions is Regular Expression Denial of Service (ReDoS). A regular expression susceptible to ReDoS will take a long time to evaluate, consuming CPU time, and is detrimental to system performance. It is not clear if this was the exact problem for Cloudflare, but certainly, something similar occurred.

Before continuing I want to emphasize this post is not a criticism of Cloudflare. While I am employed by Signal Sciences, a competitor to Cloudflare WAF, I am also a user of Cloudflare’s free tier service on my personal web site projects. Although I don’t leverage their WAF explicitly, I use Signal Sciences as my WAF. In my opinion, incidents happen and how a company responds is important. I think Cloudflare communicated and resolved quickly, as well as posted informative updates on the incident. To me, this seems be a good response so far given the unfortunate circumstances. The last piece of the response will be what changes to avoid a repeat of the incident. Regardless, the point of this post is anyone can have a bad day, and now let’s turn to more thematic issues.

The Root Cause

Taking a step back, there is a broader issue that I submit as the root cause of the incident. This is the WAF technology itself, moreover legacy WAF technology. Legacy WAF solutions primary mechanism for attack detection is regular expressions. This results in a security control that at a high risk of false positives that can break applications if in blocking mode, or overload analysts with fruitless alerts. In addition, the complexity of regular expressions end up being very difficult to maintain over time, and in the worst-case introduces the risk of a ReDoS. There is an unfortunate trade-off that must be made, either dumb it down to avoid headaches and risk or invest heavily in time and resources to ensure effectiveness, efficiency and hopefully reduce outage risk. In today’s world of the modern web, business drivers, and threats this is not a trade-off a security team should be forced to tangle with. In the long term, it’s simply unaffordable. Unfortunately, Cloudflare, as well as every other WAF technology, is primarily based on the legacy approach of regular expressions.

Another piece of data that underscores the risk is several recent CVEs related to ReDoS in the ModSecurity Core Ruleset Project. Five CVEs in total. ModSecurity has been the foundational technology in mostly all legacy WAF products. If you are using a WAF product today and are not sure of what technology it is based on, you should ask your provider. If the answer includes ModSecurity or regular expressions, be aware of the risks. Overall, this is not a great story for a security control.

Cloud Outages

Another component of risk to acknowledge with this incident, and others like it, are outages of cloud services. Now more than ever our businesses depend on cloud providers and their services. An outage for them translates to an outage for us. The dependency on cloud services is only going to increase and become more critical. On the surface, I think most users of the Internet and our sites and applications have come to terms the possibility of an outage, and even accept it to a degree as a fact of life. While such an incident is embarrassing for the provider and inconvenient to users, any outrage will subside and the incident will be forgotten in a few months if not a few weeks. For a 30 minute outage like Cloudflare’s, this will likely be the case.

However, as a business dependent on cloud services you need to understand the impact from a cost and opportunity loss perspective. Understand the risk. What are some of the impacts to consider from a risk realized event. It really depends on the context of your online business, and you should understand what is important to your business. For a low traffic volume web site, 30 minutes may not be a big deal. But what about an hour, two hours, six hours, or more? What is your tolerance for downtime? For a high traffic volume web site, 30 minutes can certainly be a big deal. Anything beyond that time frame only compounds the impact.

A few specific examples to think about are:

New user registration. If a user attempts to create a new account on your site, but can’t due to an outage they may go to a competitor. To help measure this risk you’ll need to track the average frequency of new user registrations over time. A simple example of this could be an average of 25 new user registrations per day.
Financial transactions. Specifically purchases or anything else that impacts your bottom line. To help measure this risk you’ll need to track the average number of transactions over time. To be even more specific, also track the average value of the transactions.

These are two high impact examples, but your business applications may have more. From these examples, there are two primary points I’d like to highlight. The first point is on mitigation and the second is on instrumenting your applications so you can know and understand the impact of an outage.

Mitigating the impact of a cloud outage is not so different from what you may already have planned for disaster recovery. The key here is, having a plan. In the cloud world, this could mean implementing a multi-cloud solution for your application deployments. The question to ask yourself is, can your cloud deployments on cloud provider A be seamlessly and quickly deployed to cloud provider B? This would mean you have accounts and provisioning already established and the orchestration tools that enable the seamless move. Certainly, a multi-cloud solution introduces additional cost to your cloud plans, but that should be weighed against what your tolerance is for an outage.

My second point on the instrumentation of applications is about measuring and understanding what your tolerance for an outage may be. Having the capability to instrument your applications has far more benefits than measuring the frequency of user registrations, logins, or high-risk transactions. There are very signification threat benefit and mitigation benefits to be gained with the capability. Having said that, instrumentation gives you visibility on what is important to your business as well as the data. Without that data, you cannot assess your risk accurately, which is necessary to plan appropriately for cloud outage mitigation.

Conclusion

Outage happens! It is a fact of cloud life. Cloud providers will continue to improve service robustness over time so we can expect outages to be rare overtime. However, that doesn’t mean it would be risk appropriate to rely on cloud providers to ensure the robustness of our business online. This is your responsibility to plan for. If you are using a legacy WAF technology, it is evident that you should consider a modern WAF such as Signal Sciences, which helps eliminate the risks associated with regular expression based attack detection. In addition plan for cloud resiliency, which might mean a multi-cloud solution. Finally, instrumentation of applications is crucial to obtain the visibility required to understand business risk and defend applications in the modern web.

Simple Tool for Testing CVE Mitigation in Web Apps

Px Mx — Fri, 19 Apr 2019 20:17:04 +0000

(Photo by VADIM GHIRDA/AP — source)

With Internet exposed web applications prompt mitigation of CVE (Common Vulnerabilities and Exposures) is critical. When a new CVE has been announced the response drill is all too familiar to InfoSec teams. First, evaluate the details of the CVE to determine if any of their applications are impacted. In many cases, it’s a scramble to try and discover if the vulnerable framework or code is even being used in their environment. Next, Google searching to find any proof of concept (PoC) exploit code that may have been published. Finally, once vulnerable web applications have been identified, initiate the patching process as soon as a patch is available.

Patching a CVE as quickly as possible is critical as published PoC exploit code is typically weaponized within malicious automation that is continuously scanning the Internet for vulnerable victims.

In some situations patching the vulnerability is either not an option or it could take weeks for the patch to be deployed in production. In these cases, applications are left exposed, so an alternate mitigating control is required. This often comes in the form of a new rule in the Web Application Firewall (WAF) or some other web server filter. When deploying these rules, validation of its efficacy is important (trust but verify). To perform validation there are a few options:

Vulnerability scanner — scanners generally try to determine if you are vulnerable in a passive manner, e.g. looking for version strings in headers or looking for other attributes that indicate the vulnerable code is present. For some vulnerabilities, not all, a scanner can attempt a benign exploit attempt to determine if the application is vulnerable. However, it does not actively validate if your mitigating control is effective. If your mitigating control returns a result the scanner did not expect then the scanner would report the application is not vulnerable.
PoC exploit code — PoC exploit code can be a good way to validate a mitigating control. They are crafted for the specific vulnerability and will attempt to exploit it and perform some level of exploit verification. However, PoC exploits can come from multiple sources, Github is a typical source. It’s common for single CVE to have multiple sources for a PoC exploit — spread across several projects on Github or other repositories. This can become a bit cumbersome to manage. However, Metasploit can help to consolidate the management effort as it will have exploit modules for many of the CVEs.

In addition to mitigating the risk of an attack, adding CVE specific rules to your WAF for detection purposes can be useful — even if the vulnerability is patched. The information gathered from attack attempts can be fed into your threat intelligence or anomaly detection systems.

A Simple Tool

Getting to the point of this post, I want to share a tool that aims to help with validating CVE mitigation controls (WAF rules or other server filters) you’ve implemented. The tool contains various PoC payloads (collected from various PoC exploit code on Github and other sources) to test that your mitigating control is properly detecting and blocking the attacks. The tool is not attempting to exploit the vulnerability for verification, but rather it is simulating the attack based on exploit code and expecting the mitigating control to return a specific status code. The status code is the indicator that determines the attack was detected and blocked. For example, some WAFs will return a 403 status code to indicate a block has been enforced. In addition, ease of use helps make testing repeatable for regression testing.

This simple tool is a Python script and does not have a fancy name, it’s just humbly called web-cve-tests. You can find it on Github here:

foospidy / web-cve-tests

A simple framework for sending test payloads for known web CVEs.

web-cve-tests

The goal of this tool is to send PoC payloads to verify server-side attack detection solutions. If detected, the server side should return a specified HTTP status code.

This tool is not intended to actually exploit the vulnerability or to test for the existence of the vulnerability.

Usage

Basic:

./webcve.py --url https://target-site.com

Specify detected response code (default is 403):

./webcve.py --url https://target-site.com --status-code 406

Verbose (output CVE descriptions):

./webcve.py --url https://target-site.com -v

Test a single CVE (with example output):

./webcve.py --url https://target-site.com --status-code 406 --cve CVE-2017-9791 -v
CVE-2017-9791
The Struts 1 plugin in Apache Struts 2.3.x might allow remote code execution
via a malicious field value passed in a raw message to the ActionMessage.
        Test passed (406)
        Test passed (406)
        Test passed (406)
        Test passed (406)

Test for a group of CVEs. Groups are defined in groups.json.

./webcve.py --url https://target-site.com --group struts

Test for a group type of…

View on GitHub

It currently contains tests for 26 CVEs (see the full list here), but I’ll continue to update it as often as I can, and as new CVEs impacting web applications are announced. As an open source project, contributions are always welcome. Especially, if the broader community finds this tool useful and would like to see it evolve. If you are in AppSec and are handy with the Python — hint hint ;-)

By just passing the target URL, web-cve-tests will launch all CVE tests. You can also specify a single CVE to test or a group of CVEs. For example, if you only want to test all Struts CVEs you’d pass the command line option --group struts.

Below is an example of execution and output. In this example, a status code of 406 is expected to confirm detection, a single CVE is specified for the test (CVE-2017–9791), and output is set to verbose. Each test has a variant attack payload for the one CVE.

./webcve.py --url https://target-site.com --status-code 406 --cve CVE-2017-9791 -v
CVE-2017-9791
The Struts 1 plugin in Apache Struts 2.3.x might allow remote 
code execution via a malicious field value passed in a raw message
to the ActionMessage.
        Test passed (406)
        Test passed (406)
        Test passed (406)
        Test passed (406)

Conclusion

When patching a CVE is not possible or delayed, implementing a mitigating control is critical to mitigating the risk of being exposed. Vulnerability scanners can help with indicating if the presence of the vulnerability, but they don’t necessarily verify that mitigating controls are detecting attacks. PoC exploit code can be effective at verifying if the mitigating control is working as expected, but if you need to test for numerous CVEs collecting and managing the various PoC scripts can be burdensome — although Metasploit can help reduce the burden. The web-cve-tests tool aims to provide a simple way to test CVE mitigation by leveraging PoC exploit code from various sources. This tool can be leveraged for quick testing and validation and also made part of a repeatable process for regression testing.