DEV Community: edowadohu

Beyond disaster recovery: 4 major considerations for your website

edowadohu — Thu, 03 Oct 2019 03:30:49 +0000

Disaster recovery should be at the core of every organization’s IT strategy. Any tactics implemented to keep a company online in the event of a catastrophe should account for natural and manmade disasters. Both threats have the ability to stop an organization in its tracks. The threat posed to business websites by bad actors looking to hijack functionality requires additional considerations beyond standard disaster recovery protocols.

Forty-three percent of cyber-attacks launched in 2018 targeted small businesses. That can be eye-opening for organizations with less than 250 employees.

While the likes of Target tend to be the ones who draw the big headlines and seemingly larger payouts, they often get to them by going through a smaller vendor or associate company. The target itself got hacked because bad actors gained access through the system of an HVAC company they used in the past.

The true cost of disaster recovery

Many small and medium-sized businesses (SMBs) cite cost as a reason why they don’t go further with protection measures. They rely on the false confidence that hackers will go after bigger fish. That mistake can end up crippling an SMB’s operations and potentially put them out of business.

We have previously discussed the increase in DDoS attacks against SMBs. Many of them simply cannot withstand the business disruption and loss of revenue that comes from having their business sites knocked offline for an extended period.

The Ponemon Institute estimates that cyber-attacks end up costing SMBs around $3.86 million dollars on average. Even a single hour of downtime can bring a price tag of over $100,000. What goes into the build-up of such large numbers?

Cleanup costs
Lost revenue from business disruption
Remediation costs in highly regulated industries like finance or healthcare
Regulatory fines that arise from compliance violations
Processing customer refunds and covering costs they incurred as a result of a breach

Suspending operations after a cyber-attack is a path from which many SMBs do not come back. Covering expenses like rent, payroll, insurance, and operational costs with no revenue coming in takes a toll. Sixty percent of SMBs that suffer a cyber-attack end up going under after six months.

Five additional considerations for keeping your site online

Disaster recovery is an essential aspect of avoiding downtime during catastrophic events. Companies should ensure they look out for other vital elements of their IT infrastructure when it comes to maintaining high availability of their website and company systems.

DNS accessibility

Your Domain Name System (DNS) is how readable domain names get converted into IP addresses for connection. That is why we can enter “www.google.com” into our web browser instead of strings of numbers. DNS support services form the backbone for resolving internet domain names.

Distributed denial-of-service (DDoS) attacks disrupt your domain’s DNS resolution attempts. Doing so prevents users from accessing the website. The attack also compromises your website’s responses to legitimate traffic. It can be hard to separate a DDoS attack from heavy traffic because they mimic them by using a variety of unique locations and look for actual existing records available on the domain.

A system of round-robin load balancing can keep your site from being taken offline. You should list the IP addresses of two load balancers, or reverse proxies, that are each set to balance traffic loads between your CDNs. If one becomes unreachable, a client web browser can continue to the next DNS server and attempt to locate the website.

CDN availability

Content delivery networks (CDNs) can improve website performance and reliability by caching content at the edge of the network. Using CDNs over traditional hosting solutions also prevents service interruptions, improve redundancy, and provide added security. They have a higher capacity to withstand hardware failure while conducting a higher level of traffic. Bandwidth costs also go down since your site does not have to transport as much data.

CDNs can still go down for other reasons outside of cyber-attacks. A Cloudflare outage occurred in July 2019 because of a software deployment gone wrong. It pays to have multiple CDNs available to decrease the chances of downtime for your website. You do not want your business experiencing a Cloudflare scenario because you balked at the cost of obtaining more CDNs.

Server availability
Many companies try to ensure high availability of their servers by having a load balancer on the front. Two additional servers are then connected as a backup in case one of them goes down. Incoming traffic goes through the load balancer and is given access to the designated server.

Having automated server failover can ensure that a standby server is accessed if one of the others fails or becomes compromised. The practice can keep your organization from losing critical data. Automated server failover automatically directs your requests off-site for handling, keeping your operations seamless.

Database accessibility
You want to make sure your organization always has access to your onsite and cloud data stores in an emergency. Most companies have a single database router and multiple redundant databases. They implement a “master-slave” architecture where the “slave” servers continuously duplicate content from the central “master” database. You can help ensure high availability for your data storage solutions by having the process automated.

How mlytics patched Cloudflare WAF bypass vulnerability (on our end)

edowadohu — Tue, 20 Aug 2019 09:38:48 +0000

On Oct 25, 2018, a researcher from ODS (Open Data Security) named Daniel Fariña released a blog post sharing his findings about a vulnerability in the case of Nginx on Cloudflare, which could disable the WAF leaving the companies vulnerable to cyber attacks. There’s even a video providing a very detailed explanation and demonstration of this issue.

This finding has caught our attention as we are also using Nginx to develop our own WAF (see article: Why and How mlytics Built Its Own Web Application Firewall).

What happened exactly?

We noticed that Lua in Nginx has a limitation in terms of accessibility to all the information of one request, and it can be summarized as follow :

“… a maximum of 100 request arguments are parsed by default (including those with the same name) and that additional request arguments are silently discarded to guard against potential denial of service attacks”.

Meaning which, any WAF developed on top of Nginx in this scenario will be left vulnerable if one request is using this as a loophole to keep everything undetected. If the parameters that contain threats are not being supported within the scope, it will be totally unusable.

So why do we have to patch anything?

At mlytics, we give our users the liberty to choose what CDN platforms to use, while keeping them protected via our proprietary WAF to safeguard and unify security policy across platforms. Since some of our users have enabled Cloudflare via mlytics’s Multi CDN, thus making them vulnerable to this issue.

We didn’t wait long for Cloudflare to patch the hole. We can intercept malicious requests that bypass Cloudflare’s Lua-Nginx vulnerability via a patch to the mlytics platform to keep our users protected.

It works!

We did a couple of before and after tests, here are the results:

Before patch
Test scenario: with one parameter

“curl -i ‘127.0.0.1/?txtSearch=<%21–%23cmd’ -H “Host: demo.1testfire.net” **HTTP/1.1 403 Forbidden** Server: nginx Date: Thu, 13 Dec 2018 07:08:05 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Cache-Control: no-cache <!DOCTYPE html><html lang=”en”><head><meta charset=”UTF-8″><title>Error Page</title><link rel=”stylesheet” type=”text/css” href=”__assets/css/style.css”><link href=”https://fonts.googleapis.com/css?family=Raleway” rel=”stylesheet”></head><body><div class=”wrapper”><h1>ACCESS DENIED<span>Your request to access demo.1testfire.net was denied</span></h1><p class=”error_info”><span>Incident ID </span>31c75a46e100079d1449f5e4db85d6de</p><p class=”error_info”><span>Your IP </span>127.0.0.1</p><img src=”__assets/img/process_img.png”><div class=”next_Step”><p><span>What happened ?</span>The website you are trying to access is protected against cyber attacks. Your recent action or behavior was flagged as suspicious. Further access to the web server has been denied.</p> <p><span>What can I do ?</span>Please try again in a few minutes. Or, you can directly contact the site owner within Event ID indicated and a description of what you were doing before you were denied access.</p></div><span clas=”copyright”>Powered by mlytics.com</span></div></body></html>“

Test scenario: with a0-a9, 10*10, a total of 100 parameters, and then add the 101st parameters to the command injection payload

“curl -i ‘127.0.0.1/?a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&<%21–%23cmd’ -H “Host: demo.1testfire.net” **HTTP/1.1 200 OK** Server: nginx Date: Thu, 13 Dec 2018 07:20:29 GMT Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive Cache-Control: no-cache Pragma: no-cache Expires: -1 X-AspNet-Version: 2.0.50727 Set-Cookie: ASP.NET_SessionId=2vb4y5453apg1cvpakfjigip; path=/; HttpOnly Set-Cookie: amSessionId=6207394219; path=/ X-Powered-By: ASP.NET <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”> <html xmlns=”http://www.w3.org/1999/xhtml” xml:lang=”en” > <head id=”_ctl0__ctl0_head”><title>……………….“

As you can tell before our patch, a request within 100 parameters can be easily blocked by the Cloudflare WAF. But once it goes above 100 and hit its 101st parameters, Cloudflare WAF became inactive and let it pass.

After patch
Test scenario: with one parameter

“curl -i ‘127.0.0.1/?txtSearch=<%21–%23cmd’ -H “Host: demo.1testfire.net” **HTTP/1.1 403 Forbidden** Server: nginx Date: Thu, 13 Dec 2018 07:08:05 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Cache-Control: no-cache <!DOCTYPE html><html lang=”en”><head><meta charset=”UTF-8″><title>Error Page</title><link rel=”stylesheet” type=”text/css” href=”__assets/css/style.css”><link href=”https://fonts.googleapis.com/css?family=Raleway” rel=”stylesheet”></head><body><div class=”wrapper”><h1>ACCESS DENIED<span>Your request to access demo.1testfire.net was denied</span></h1><p class=”error_info”><span>Incident ID </span>31c75a46e100079d1449f5e4db85d6de</p><p class=”error_info”><span>Your IP </span>127.0.0.1</p><img src=”__assets/img/process_img.png”><div class=”next_Step”><p><span>What happened ?</span>The website you are trying to access is protected against cyber attacks. Your recent action or behavior was flagged as suspicious. Further access to the web server has been denied.</p> <p><span>What can I do ?</span>Please try again in a few minutes. Or, you can directly contact the site owner within Event ID indicated and a description of what you were doing before you were denied access.</p></div><span class=”copyright”>Powered by mlytics.com</span></div></body></html>“

Test scenario: with a0-a9, 10*10, a total of 100 parameters, and then add the 101st parameters to the command injection Payload

“curl -i ‘127.0.0.1/?a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a0=0&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a1=1&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a2=2&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a3=3&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a4=4&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a5=5&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a6=6&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a7=7&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a8=8&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&a9=9&<%21–%23cmd’ -H “Host: demo.1testfire.net” HTTP/1.1 403 Forbidden Server: nginx Date: Thu, 13 Dec 2018 07:18:51 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive Cache-Control: no-cache <!DOCTYPE html><html lang=”en”><head><meta charset=”UTF-8″><title>Error Page</title><link rel=”stylesheet” type=”text/css” href=”__assets/css/style.css”><link href=”https://fonts.googleapis.com/css?family=Raleway” rel=”stylesheet”></head><body><div class=”wrapper”><h1>ACCESS DENIED<span>Your request to access demo.1testfire.net was denied</span></h1><p class=”error_info”><span>Incident ID </span>-</p><p class=”error_info”><span>Your IP </span></p><img src=”__assets/img/process_img.png”><div class=”next_Step”><p><span>What happened ?</span>The website you are trying to access is protected against cyber attacks. Your recent action or behavior was flagged as suspicious. Further access to the web server has been denied.</p> <p><span>What can I do ?</span>Please try again in a few minutes. Or, you can directly contact the site owner within Event ID indicated and a description of what you were doing before you were denied access.</p></div><span class=”copyright”>Powered by mlytics.com</span></div></body></html>“

With mlytics patch to the platform, the same request got rejected by our platform despite nothing has changed on Cloudflare’s end.

How we mitigated one of the largest CC (DDoS) attacks

edowadohu — Tue, 20 Aug 2019 09:19:46 +0000

In the first quarter of 2019, one of our customers came to us with a problem: their website was under attack.

No matter what they tried to get around the problem, their site kept crashing. The kind of attack being used on this particular client is known as a distributed denial of service (DDoS) attack, specifically a challenge collapsar (CC) attack.

In this case study, we’ll go into what a DDoS attack is in more detail, define the specific type of DDoS attack being sued here (CC attack) and explain how we were able to help our client mitigate the problem.

What is a CC attack?

There are many types of DDoS attacks in the wild, and a CC attack is classified as a DDoS application layer attack. When a CC attack occurs, multiple HTTP requests are sent simultaneously to overload the server, since the Uniform Resource Identifiers (URIs) has to undertake complicated algorithms or database operations to exhaust the resources of the targeted web server.

Imagine multiple missile trucks firing at your website

The attacker simulates a scenario where a large number of users are accessing pages all the time. Because the accessed pages require a lot of data operations (consuming many CPU resources), the CPU usage is kept at the 100% level until normal access requests are blocked.

See all types of DDoS attacks here.

Fun fact: CC attacks got the name “Challenge Collapsar” attacks in 2004 when a Chinese hacker who went by the monicker Kiki invented a hacking tool called Collapsar capable of blitzing a server with HTTP requests.

One of the largest CC (DDoS) attacks we’ve seen in a while

Our customer tried everything they could think of to mitigate the attack: they tried to solve the problem via in-house DDoS mitigation and used content delivery network (CDN) services and providers who claimed they could help mitigate DDoS attacks. But each attempt failed, and their site would go down after just a few minutes when relaunched.

When the customer came to us to help solve their problem, we immediately took action to mitigate the attack. Upon investigation, we were surprised at the massive scale of this DDoS attack: 300 million requests were being sent to our customer’s server every minute, and the hacker was utilizing “user mimicking,” a practice which masks the attack traffic as normal user traffic distributed across multiple IP addresses.

Approximately 1.3 Tbps of network attack

Approximately 300M QPM of application attack

We were able to mitigate the attack and get our customer’s site up and running again, even during such a brute-force CC attack.

How we mitigated the attack

We used a two-pronged method of DDoS protection to mitigate this particular CC attack:

Geo-restriction: We were able to restrict incoming traffic to specific regions by securing traffic from the main user base’s countries and regions, as well as preventing incoming traffic from known “attack regions” like Russia, Ukraine, and India.
Enabling browser-based challenge: Our web application firewall (WAF) lets us use challenge-based algorithms to filter out CC attack bots. Built on global public cloud infrastructure, we can leverage that computing power via Multi CDN to autoscale our defenses proportionate to the attack. It’s that power that let us fend off a 300-million request per minute CC attack.

In order to implement this strategy, we needed a solid, powerful infrastructure. mlytics uses Multi CDN which combine the power of over 2,300 PoPs to give us a 2,600 Tbps capacity. That gives us a global network with the power of multi terabit-per-second capacity with which to fend off DDoS attacks.

By combining effective filtering with the power of our network, we were able to help our customer absorb even that powerful CC attack and get their site back up and running.