DEV Community: Sqreen

Security leader interview: Adam Surak on approaching security at Algolia

Sqreen — Mon, 20 May 2019 22:43:44 +0000

Today, we have a new entry in our security leader interview series. Recently, we sat down with Adam Surak, Director of Infrastructure and Security at Algolia, to discuss his approach to security and what he’s learned as far as security goes throughout his career. We wanted to share the great insights that came out of our conversation.

Tell me a little bit about yourself and how you got to where you are at Algolia

My name is Adam Surak. I am the Director of Infrastructure and Security at Algolia. I originally joined Algolia in 2014 as an SRE focused on automating infrastructure. At the time, Algolia was 8 people. Now it’s 350, and I manage a team of 22, so quite a bit of growth!

Prior to Algolia, I was a teacher and researcher at a technical university in Finland.

What is your history with security?

Probably my original and most fun experience with security was back in high school. I spent some time developing a web game. At the time, it was a popular thing to try and hack web games, so I had to spend a lot of time figuring out how to protect it.

Later on, at my job at the university in Finland, I worked with several Russians who had worked in state security in Russia. They were very strict and focused on security. Whenever we developed something, they took the approach of asking “how can we hack it?” This was my first exposure to this angle of security; this way of looking at things.

How have you approached security at Algolia so far?

Over the years at Algolia as we grew, we started to have real security requirements. Our primary problem, from a security point of view, is that to make Algolia’s service work, we need to have copies of our customers’ data, and we need to be able to read it, so we can’t just encrypt it at the edge and never look at it. This opens up a lot of potential security risk. As a result, we started to focus on security early on in the company’s life.

In early 2015, I was tagged to take on security for Algolia, as I had the deepest knowledge at the time. Now in 2019, I have a team of 3 security engineers, compliance lead, and support of an experienced legal team. We focus on security, compliance, and activities like gap analysis and SOC2 audits.

One thing we started doing super early on was focusing on building a habit around securing personal devices. We had a lot of people coming in and out of the office, but people frequently left their laptops and such unlocked. We implemented a “shame channel” in Slack, so that if you catch someone’s device unlocked, you post in the shame channel on their Slack account. It’s a lighthearted way of keeping people accountable and building the habit around keeping their devices secure. Nowadays, people tell me they even lock their computers when they’re alone at home and get up to go to the bathroom! So the habit has really caught on.

What is your philosophy on security?

I would sum it up by saying that security shouldn’t be something that gets patched on top. I truly believe that security and compliance need to be automated. A lot of things that we’re doing with security can be written as code. Historically, security engineers haven’t been looked at as software engineers, but they need to be today. Their job is to automate. The scale is too large to check everything manually.

Don’t reinvent the wheel. There are a lot of tools that have figured an aspect of security out. Take advantage of them.

Another thing I believe is of the utmost importance is finding ways to hardcode security into software. In the past, a lot of security has just been policies that no one wanted to follow. Today, these policies need to be brought into the software. One example is with passwords. Nowadays, you prevent people from having an insecure password in software. You force them to enter a secure one or else they can’t sign up or log on. We don’t do that with other security issues, but we should.

Security needs to be brought closer to developers. At Algolia, the security team sits down with developers and does a PoC of the code they want to create. They look for potential weak points and security risks together. You have to find a level of cooperation, and have the teams work together to solve problems. If the security team is just calling out risks or compliance issues and walking away, you won’t end up with a strong security situation.

How have you scaled and evolved security at Algolia?

Scaling security is a tricky question, because it’s rarely possible to scale the security team at the pace that company scales. That has been the case for us at Algolia.

In 2015, one of our customers requested that we do a pentest. It was our first one. When the report came back, they called out a couple things that didn’t matter, and a couple things we didn’t originally think were too important, such as our SSH board being exposed to the world (since we had built protections on top of it). Eventually, we realized that some of these items were worth fixing and it’s strengthened our security.

At the end of 2015, we felt confident enough to open up a bug bounty program, in invitation-only mode. We were 25 people at Algolia at the time. We invited a lot of people and eventually felt like we had caught most things, so we opened it up to the public.

It was the biggest security surprise in our company history. After a couple days, everything blew up. We got flooded with alerts and emails, and things started breaking. It took us two months to recover. There were a huge number of issues!

It’s much smoother these days, since we caught up on our security debt from the first effort. It also has another important effect: when you’re on the public record, you can’t rely on “security by obscurity.” It adds a level of pressure. When people internally are saying something is an issue, it really helps to also have someone external say the same thing. It makes it real and makes you want to fix it, rather than fall prey to the belief that the issue is too small and no one will find it or be bothered by it.

Don’t treat security like an afterthought. Invest in it early. The longer you delay, the bigger your security debt becomes.

We’ve also tried to bring about a security mindset at Algolia. Security people think differently. They’re not focused on what a feature does; they care about the risk and how they could be exposed. This mindset tries to do away with the natural inclination to be protective of your work and shield it from a security eye. By paying attention to security and showing people the impact of an exploit (by exploiting it internally for example), you can really communicate the importance of security in a way that sticks with people. It makes the impact real.

What are the challenges you see blocking your ideal state of security today?

The tools out there aren’t focusing enough on the signal to noise ratio. We need to shift that mindset and push tools towards only outputting actionable information. A lot of security tools output absolutely everything, so the real signal is buried in noise. This is super discouraging to the user. It would be much better to start narrow with the outputs and incrementally improve.

Another thing I see is the fact that security is flooded with massive amounts of data. Multiple layers, multiple applications, multiple stacks. The volume of data is so massive that security teams are pushed towards anomaly detection — just looking for outliers.

Another challenge is around vulnerable dependencies. In an ideal world, all dependencies would be kept up-to-date. In reality however, it’s not always possible. Sometimes, a thing you rely on gets deprecated in a new version or the new version breaks something in your flow, and you’re forced to stick with the old version. As a result, you need to be very aware of which CVEs you’re exposed to so you can find ways to wrap protection around them. You also need to decide if these CVEs are actual risks to you and your business. They need to be exploitable and something that could have an impact.

Security needs to move towards actionability everywhere. We need clear signals and clear recommendations of actions to take based on actual risks that could have an impact on your business.

Do you have any advice for companies or people beginning their own security journey?

If you’re thinking about starting a public bug bounty program, pause and talk to someone who’s done it first! It’s a much bigger event than it seems!

Really though, here are some pieces of advice based on what I’ve learned along the way:

Don’t treat security like an afterthought. Invest in it early. The longer you delay, the bigger your security debt becomes. I knew a company that ignored security for years before finally doing a pentest. The report they received was 100 pages long. They had to stall work for 3-6 months just to take care of the major issues.
Look at automated tools, especially early on when you don’t have to time to look at all the issues that crop up manually. You likely don’t need to be resilient against big state actors; you just need to protect against competitors and script kiddies at first. The level of security you need depends on the level of threat you can reasonably expect. For most businesses, protecting your code and your applications is enough in the beginning.
Don’t reinvent the wheel. There are a lot of tools that have figured an aspect of security out. Take advantage of them. Even outside of security tools themselves, you should leverage other companies’ security teams. For example, leverage Slack’s security capabilities rather than creating your own messaging app.
Finally, ask people in the industry for help! Despite some reputation to the contrary, most people in the security space are happy to share their knowledge and experiences with you. Ask them for coffee or to sit on your board of advisors. Involve them, and they’ll be happy to help.

Thanks so much to Adam for his time. If you’d like to read more technical and security leader interviews, check out our conversation with Ricardo Félix on scaling DevOps at Uniplaces.

The post Security leader interview: Adam Surak on approaching security at Algolia appeared first on Sqreen Blog | Modern Application Security.

How to debug memory leaks in a Node.js application on Heroku

Sqreen — Thu, 18 Apr 2019 16:25:35 +0000

Debugging memory leaks is rarely a piece of cake, especially when they only happen in production. The best way I’ve found to debug memory leaks in a Node.js application on Heroku is to analyze heap dumps.

Obtaining such heap dumps in production can be challenging, as it might be hard to connect remotely to a production instance with the debugger.

In this article, we will go through the steps needed to obtain and analyze heap dumps from a running Heroku dyno. This method will also work on other platforms as long as it is possible to perform similar operations.

To obtain the heap dump we need to:

Ensure the Node.js process has a debugger listening
Connect Chrome dev tools to the Node.js process
Collect the heap dump and download it locally

Enabling the Node.js inspector

Before we can analyze anything, we need to ensure that we have a debugger listening. There are two ways to enable the inspector on a Node.js process:

Solution 1: Changing the startup command

By default, Heroku starts a Node.js application by running npm start. Usually, this calls a script defined in the package.json of the application:

Changing this script to add the --inspect (as documented here) flag will start the instances of the application with a debugger listening on a port that will be specified in the logs:

In total, this is what it will look like when you implement this solution.

What changing the startup command looks like Heroku logs

Solution 2: Changing the process state through SSH

Solution 1 is the easiest way to enable an inspector in Node.js, but there are situations in which you can’t or won’t want to enable it. For example, you might not have access to the source code of the application and therefore can’t change the startup script. Or maybe you don’t want to change the state of all your production dynos and deploy your application only for debugging.

Fortunately, there is a way to send a signal to the process to enable a debugger session.

In order to do so, you will need the Heroku CLI to connect to the dyno through an SSH connection.

For all following Heroku commands, you might need to add the --app <app_name> flag to tell the CLI which application to connect to. Also, by default, the CLI will connect to the dyno named web.1 and you might want to change that through the command line (see documentation).

First, let’s connect to the dyno (Heroku might need to restart the dyno at this point):

Then, we need to identify the PID of the Node.js process:

In our case, the process started with node bin/www has the PID 69, we will now send a signal to the process to let it know we need it to enable its debugger:

As you can see, we have sent the USR1 signal to the process to change its state (as documented on this page).

This is confirmed through the application’s logs on Heroku:

Changing the process state through signals

Attaching debugging tools to a Node.js process

In order to attach the debugging tools to our Node.js process, we need to make the WebSocket used by the debugger accessible on our local machine.

To do that, we first need to identify the port we need to forward. This can be found in the logs of the application:

In our case, this is the port 9229.

To forward the port locally, let’s use the Heroku CLI:

When port forwarding is established, we just need to open Chrome DevTools (going to chrome://inspect on Chrome) and after a few seconds, a target should be displayed under “Remote targets.”

If the target does not appear, make sure the port used is listed when clicking on “Configure.”

Collecting the heap dump and reading it

Now it’s time to collect and read the heap dump. First, click on the “inspect” link. This will open a new window with different tabs.

Find the “Memory” one — you should be prompted with the following window:

Click on “Take snapshot.” A new file will appear in the left hand side panel. Clicking on it will display the content of the heap:

In this view, objects are sorted by constructor. For the purpose of this walkthrough, I have introduced a memory leak in this application by creating an instance of the Access class for each request. This instance keeps a reference to the current HTTP requests and is never cleaned:

You can see for yourself that this indeed leaks in the application.

To detect constructors that have the biggest memory impact, let’s sort the items of this view by “Retained size” (You can learn more about these terms on Chrome’s website).

You can see that 24% of the process memory is held by these objects.

Now let’s look at how to identify where the leak is happening.

When expanding the list of the constructor, we can see all instances of this class. By selecting one of these instances, the list of retainers of this object is displayed:

In our case, the allAccesses set is clearly identified as the bad actor! With the location of the memory leak identified, we have everything we need to go off and fix it.

A few tips for debugging memory leaks in Node.js

Use the compare view

When suspecting a memory leak, you might want to take two separate heap dumps with a few minutes between them. Then, using the “comparison view”, you can identify which elements have been created between the snapshots.

Use constructors and classes in the code

As shown in the article, when reading the heap dump, elements are grouped by their constructor.

Using more than just classes in your code will make it more readable (and arguably more performant, but that’s probably a topic for another article). It will save you so much time when hunting for a memory leak. Do it — future you will be grateful.

Trigger a garbage collection before collecting the snapshot

At the top left hand side of this screen, there’s a little bin picture. Clicking on it will trigger a garbage collection in the application. Doing this before collecting a memory snapshot will actually remove elements that are not leaking and therefore could help save you time when browsing the heap content.

Conclusion

In this article, we’ve taken a look at how to debug memory leaks in a Node.js process running on Heroku by connecting and using a debugger. Feel free to contact me on Twitter if you have any questions or if you want to share your own tips with me!

If you’re looking for next steps or a more advanced way to debug memory leaks in Node.js in Heroku, try this: Since the Heroku CLI is written with Node.js, you could write an automated tool to perform the collection and start analyzing heap dumps.

The post How to debug memory leaks in a Node.js application on Heroku appeared first on Sqreen Blog | Modern Application Security.

How to fully leverage your pentest

Sqreen — Thu, 28 Mar 2019 16:05:54 +0000

Before co-founding Sqreen, I spent years stressing applications both from the outside (as a security consultant) and from the inside (as part of Apple’s Red Team). As a result, I’m often asked by fellow CTOs what the best practices are in terms of SaaS security and penetration tests (pentests). So this article will shed light on pentest best practices and some mistakes you should avoid.

There are plenty of ways to proactively test an application’s security. One of the more common methods is performing penetration tests on your application. A pentest is a way to reveal security issues in your application and infrastructure from a neutral third party in a way that you can leverage to positively affect the whole company.

There are plenty of resources out there to help you prepare for a pentest. This article focuses on the actionable learnings a pentest should provide to you, beyond the pages of the report provided by the pentesters.

The 3 axes of a pentest

The 3 learning axes of a pentest

The learnings you can get out of a penetration test can be grouped into 3 dimensions.

the technical learnings:
- the state of your code security
- the state of your infrastructure security
- the quality of the monitoring and security alerting in place
the organizational learnings:
- how new services or infrastructure changes are handled (e.g. moving to the cloud)
- whether processes and ownership are clearly identified
- the effectiveness of the processes in place
the human learnings:
- the team’s reaction when security issues are reported
- the way the company (hierarchy, stakeholders, etc.) handles security reports.

The technical report and the discussion with the auditors will mostly tell you about the technical axis. It is up to you, however, to carefully prepare your pentest in order to walk away with key insights and clear action items around the human and organizational dimensions.

A pentest also provides a way to assess how your security has evolved over time. Every penetration test should be followed by some days of validation tests – to ensure that the items you found and planned to fix have indeed been fixed. You should also provide previous test reports to the security auditors as a starting point to find initial vulnerabilities, iterate on the existing ones, and ensure they were fixed (if they’re present again, it means a regression occurred).

The technical learnings

A pentest will very likely give you new technical insight into your infrastructure and application. It’s important to thoroughly analyze how the technical side of your app responds to the pentesters, and how the tools you’ve set up to monitor and protect your app react. The report you get from the pentesters will give you a great overview, but you know your environment better than anyone, and there’s a lot to learn as well from how it responds during the pentest.

Prior to the pentest

Before the pentest begins, you and your dev team should review the backlog you have around security. Do you have a security tag in your bug tracker? How many tickets are tagged with security? How old are they?

No security tickets in the backlog most likely denotes a lack of attention on the security front. Otherwise, check how many have been solved in the past months. If the answer is none, it probably means you haven’t been prioritizing security enough!

Also, ensure your team is up to date on their technical security knowledge: does everyone knows what an XSS is? An injection? Same Origin Policy? CSRF? Content Security Policy? If your team has too many gaps, then you will need to start with updating your team’s knowledge prior to beginning any kind of fixes after the pentest concludes.

During the pentest

The quality and effectiveness of your monitoring and alerting tools is one factor to check when you’re doing a pentest. There are plenty of things a team monitors that may trigger because of a pentest. An exception manager, such as Sentry, will have unusual activity during most penetration tests that your dev team will need to interpret.

You should also check your performance monitoring tool (such as New Relic), which can highlight slow parts of the application during the pentest. The tests may slow down your application and put pressure on some specific parts of your infrastructure (e.g. a database or a microservice). This can lead to the detection of potential applicative DOS, and require specific optimization or rate limiting in order to prevent exhausting it. Most penetration tests do not look for DOS since they focus on testing the production: that’s something you need to look for yourself.

Your log monitoring solution will be a good source of information as well and can also tell you if your application is getting too slow and starts to answer with timeouts or errors. It’s worth checking your logs to see if you have issues in this space.

Eventually, your security tools will generate alerts regarding the attack. Make sure these alerts came from the testers (based on the IP address, the user names, etc.…) and take screenshots. The testers are gonna be pleasantly surprised you managed to catch them.

All of this information will most likely be unnoticed by a security researcher performing a test on your application, because they only have an external point of view on your system. It is up to you to be reactive when these kind of alerts occur and interpret them appropriately.

Refrain from fixing anything the testers may report to you until after the pentest fully wraps up. Doing so would prevent them from fully understanding and exploiting what they found, and also from finding more vulnerabilities in that vein.

After the pentest

The report that the pentesters deliver to you at the end of the test will cover any technical vulnerabilities that they’ve uncovered. It is up to you and your team to then prioritize and implement fixes. Many pentests come with a validation period, which is a time during which people can just ask their security contractor to ensure the reported weaknesses have been validated. Using a clear mapping between the pentest and your tickets will help you organize and prioritize this accordingly.

Take the time to speak in depth with the pentesters about what they’ve found in order to get the most you can out of the exercise. Pentesting companies often offer training, and organizing half a day of training on a particularly hard-to-understand vulnerability could be especially useful to level up your developers and get them interested in the subject.

Additionally, the report serves as a point-in-time understanding of the state of your infrastructure security and the state of your code security. It will also help you to take a step back and figure out where the gap areas are in your team. Should you train your team on a given family of vulnerabilities? Improve your processes to have a bigger security focus when writing specifications? Hire someone to challenge your team on their security point of view? Hire someone who could be a security champion? The pentest report can help you decide answers to questions like these.

tl;dr: actions for the technical axis

Understand your security issue backlog and the state of security knowledge on your team
Check that your monitoring solutions are working and are properly used by your developers
Take the time to fully understand the report that the pentesters produce, and take concrete steps to improve your security knowledge on the team and your security within your applications

The organizational learnings

Pentests can give you insight into your organization as well. The way your security processes work (or don’t) in practice and the way your team functions as a unit are all key things to watch for and determine what needs improvement.

Prior to the pentest

This is the time to review your security processes. Ensure the responsibilities for your various processes are well-defined and clearly known. You should know who to alert when security vulnerabilities are found. It is not infrequent that penetration tests highlight existing vulnerabilities that are currently being exploited , and that may require a rapid response – you might need to shutdown a service while the vulnerability is being corrected. In this case make sure you know how to react and how to articulate and route this information within your company – this may involve the legal department as well as the communications department, particularly in the case of public companies.

During the pentest

How the team responds to security alerts is a key element of an efficient and effective security response. Is the team trained to do so? Was a process ready for the person on call to respond to this? These processes don’t get triggered very often, so is the team aware of how to find and implement this process? Have all the security alerts sent by your security tools been handed over to the appropriate team? For instance, a vulnerability lying in the code should go to the developer team to fix, and a training session could be conducted to ensure all engineers are now aware of this specific security threat. Also, any new developer joining the team needs to be given this training as well.

After the pentest

Some of the vulnerabilities discovered during the penetration test will come from a failure to enforce some processes – or from a lack of process. The initial fix is always technical, such as changing a default password or upgrading a library. But the root cause of the issue often lies in a missing or incomplete process (or an imperfect automatization).

This can be especially true when new technologies are at stake. The security specifics of a new database system, or a new cloud provider may remain fuzzy to your team, especially from a security point of view. In this case, training should be conducted so your team better understands these technical elements, and can ensure they are correctly configured. This also includes the need to improve the process of adding new technology to your infrastructure: engineers should be trained to reach a decent level of knowledge on the technology from a security point of view. A security review of any new family of elements can also be enforced.

tl;dr: actions for the organizational axis

check you have procedures in place in case of a critical flaw discovery
check that your processes have clear ownership defined
test your processes to ensure that people know how to apply them

The human learnings

Pentests can tell you a lot about the individuals that you have on your team as well. Their reactions and responses can communicate a lot about their preparedness and mindset towards security.

Before the pentest: accepting the idea

For many technical people, whose job it is to build and operate systems, knowing that a penetration test is being run on what they built can be seen as an offense to their skills and intelligence. It can also be understood as a lack of trust from whoever ordered the tests.

Of course, this is a spontaneous reaction, driven by their ego, that may not last. The team will soon realize that a penetration test has two positive outcomes:

First, it is an excellent opportunity to learn, since most security professionals will detail their reports in a pedagogical way. Learning is one of the most shared motivations amongst IT professionals.
Second, it is an excellent opportunity to prioritize leftover tasks that generated security debt – which is a subset of technical debt.

The takeaway here is that the way you present the pentest to your team matters. Do not hide behind a compliance excuse as the reason for the tests, but rather clearly leverage these positive points to have the team welcome the security tests.

To ease the impact of the pentest on your team’s psyche, tell them in advance that a test is coming and promise that they will know the timeframe (e.g. 3-6 months out). Giving them a range rather than an exact date also creates an opportunity for them to get in the habit of thinking about security in their sprints, as well as improving or checking the alerting system for security issues. The team is often aware of security issues that have been long standing in the backlog, so communicating ahead gives them the chance to get around to fixing those issues.

During the pentest: monitoring & reacting to attacks

While the tests are ongoing, the team will be warned about the issues by your security alerting tools, if you have one in place (which you should). Critical issues will need immediate response (depending on your agreement with the security auditor) and will be handled by someone from the team.

Pay extra attention to the human side of these security issues. They will have stronger psychological impact to your team than usual bugs do, since they are proven to have security impact, and will be highlighted in the security auditor’s report that may flow all the way through the company’s hierarchy. This may reveal tensions and finger-pointing across team members. It is the right time to remind everyone about the benevolent motto: everyone is doing his or her best. If mistakes are made, they need to be understood and the team should act accordingly to ensure they won’t happen again. The important thing is correcting the bugs, not assigning blame, so this whole process should be conducted blamelessly.

After the pentest

Once the security audit is over, the team needs to debrief about the results. What was detected during the tests needs to be consigned in a shared document and enriched by everyone. Getting a broad team involved will make the results stronger.

Security successes (e.g. successfully detecting attacks, the parts of the system that the pentesters weren’t able to break, etc.) need to be highlighted and applauded, and failures (e.g. vulnerabilities detected and exploited, weaknesses discovered) have to be added to your bug tracking system, with a specific way to list them (with a tag such as “Q2 penetration test”). As with any other issues, these ones need to be prioritized. Most security consultants will rank their findings to help you prioritize what to fix, but they most often do not have the best understanding of your business. That means you need to go deeper with your team (and maybe the security consultants if an issue outcome is not perfectly clear) in order to properly rate and prioritize them.

Each issue found (e.g. MongoDB injection in the Company controller) needs to be generalized, multiple times. For example, “if we have injections in this controller, we may have some in other controllers”. “If we have injections, maybe we are not that good at sanitizing inputs, let’s check user inputs”. And so on. This will directly leverage your team’s problem solving skills.

Eventually, analyze your company’s reaction up and down the chain of command to the bad news that came up during the tests. Security is a sensitive topic that may put people in what they feel is a sensitive position, and have them react with more stress than usual. Good faith, honesty, and transparency are critical values for employees in order to have them proactively report security issues – rather than burying them.They have to be listened to and encouraged in order to bring these issues up. This means more than just stating values. People will read how everyone reacts during the pentest and internalize that.

A typical reaction when security vulnerabilities are found goes like this: “this is indeed an XSS, but it isn’t exploitable because only the user can do it to himself.” This reaction is understandable: if the vulnerability cannot be exploited, why would we fix it?

Let’s consider this vulnerability as a regular bug. What do you do with regular bugs? You fix the ones that are quick to fix, or can harm you, or can endanger your application. You also learn the lesson and make sure these kind of bugs do not occur again. Security bugs have to be considered the same way. If it occurred once, it may occur again, and if the part of the application where the vulnerability lies is later reused some place else, the bug will spread. In this case, its exploitability will have a different impact.

It is a regular bug, but one that has security consequences. That merits a different level of attention from your team.

tl;dr: actions for the human axis

Take the time to talk with your team about the pentest before it happens and present it to them the right way
Pay attention to the psychological side of your team during the pentest. Ensure that everyone knows that whatever is uncovered has nothing to do with blame
Take the time to debrief fully with your team, and prioritize and log the security issues found in a fair way

Need some help?

If you’re interested in learning more about maximizing your pentest, we put together a Pentest Best Practices Checklist. Check it out!

Pentesting can give you an excellent point-in-time understanding of your security and show you the critical issues that you need to fix. Sqreen can help you solve these issues with real-time monitoring and protection, and actionable alerting. They work great together! Try it free today or sign up for a demo.

The post How to fully leverage your pentest appeared first on Sqreen Blog | Modern Application Security.

The ultimate PHP security checklist

Sqreen — Thu, 23 Aug 2018 11:25:04 +0000

Read the full checklist here

Damn, but security is hard. It’s not always obvious what needs doing, and the payoffs of good security are at best obscure. Who is surprised when it falls off our priority lists?

This security checklist aims to give developers a list of PHP security best practices they can follow to help improve the security of their code.

Here is a selection of some of the security checklist items:

Filter and Validate All Data

Regardless of where the data comes from, whether that’s a configuration file, server environment, GET and POST, or anywhere else, do not trust it. Filter and validate it! Do this by using one of the available libraries, such as zend-inputfilter.

Read more:

Use Parameterized Queries

To avoid SQL injection attacks, never concatenate or interpolate SQL strings with external data. Use parameterized queries instead and prepared statements. These can be used with vendor-specific libraries or by using PDO.

Learn more:

Set open_basedir

The open_basedir directive limits the files that PHP can access to the filesystem from the open_basedir directory and downward. No files or directories outside of that directory can be accessed. That way, if malicious users attempt to access sensitive files, such as /etc/passwd, access will be denied.

Read more:

Check Your SSL / TLS Configurations

Ensure that your server’s SSL/TLS configuration is up to date and correctly configured, and isn’t using weak ciphers, outdated versions of TLS, valid security certificates without weak keys, etc, by scanning it regularly.

Read more:

Check out the full checklist here

An in-depth look at CVE-2018-8778 or why integer overflows are still a thing!

Sqreen — Fri, 30 Mar 2018 02:16:27 +0000

A new exciting vulnerability (yes sorry, we easily get excited about these things! 😜) has been released in Ruby. CVE-2018-8778 is a Buffer under-read that is triggered by String#unpack. In this article, we will do a deep dive into the vulnerability, show how to exploit it and how to mitigate it.

What’s a buffer under-read?

Nearly all meaningful computing we do is done on data structures (object) that are stored in memory.

Each object has a defined size and a layout of fields in memory. So someone looking at the memory can see our object flattened as binary data of a given length (zeros & ones). This is what we call a buffer (a zone of memory). As a developer, when operating on our object, we should work within a given buffer and shouldn’t read/write before/after our object.

Ruby allocates data on what is called the heap. That’s a memory space (another one is the stack). Nearly every Ruby object will land there.

So a buffer under-read is a vulnerability that allows attackers to access memory locations before the targeted buffer. This typically occurs when the pointer or its index is decremented to a position before the buffer.

It is a critical vulnerability, but the severity really depends on the data that your application handles. It may result in the exposure of sensitive information or possibly a crash. You could potentially read data like tokens, database credentials, session cookies or even a transiting credit card number.

So what the heck is CVE-2018-8778?

Here is the announcement of the vulnerability. To better understand it let’s first dig into the Ruby source code of the fixing commit here (yeah OK it’s an SVN revision, Ruby history predates Git!).

The vulnerability is located inside the String#unpack method. This method decodes str according to the provided string format, returning an array of each value extracted (You can read more about it on RubyDoc). The format string consists of a sequence of single-character directives (numbers, “*”, “_” or “!”) and can specify the position of the data being parsed by the specifier @. That’s where the issue lies.

From the non-regression test, we can see that one just need to use a specific format to trigger this. This format string actually is like a mini program. The string “@42C10” decodes as: skip 42 bytes then decode 10 8-bit integers.

The issue here is that the offset is poorly validated. If a significant number is passed with @, the number is treated as a negative value, and unpack skips a negative amount of bytes. This is where the out-of-buffer read occurs. So an attacker could use this to read sensitive data on the heap.

The poorly validated offset is a classic mistake called integer overflow. When using signed integers, trying to decode what would be a huge unsigned integer value, the decoded value will be a negative number. Here it gives us a way to have negative offsets. On a related note, the first Ariane 5 crash was triggered by this… Source.

How does this integer overflow happen here?

String#unpack is actually defined in Ruby core source code in the C programming language. As we can see in the remediation commit the offset that is expressed as a string (a char * in C) has to be translated to an integer value. For this, Ruby uses a macro called STRTOUL which in turn calls ruby_strtoul (they are defined in ruby.h). As the name seems to prefigure this will output an unsigned long integer.

unsigned long ruby_strtoul(const char *str, char **endptr, int base);

Until here, no issue, the string “18446744073709551416” is correctly decoded to the long integer 18446744073709551416. Yet this value is stored in len which is declared as a long, a signed number. Doing this casts the unsigned number to signed number and 18446744073709551416 becomes -200.

Here are the offending pieces together:

unsigned long ruby_strtoul(const char *str, char **endptr, int base);
#define STRTOUL(str, endptr, base) (ruby_strtoul((str), (endptr), (base)))
// ....
static VALUE
pack_pack(int argc, VALUE *argv, VALUE ary)
{
// ...
    long len, idx, plen;

// ....

    else if (ISDIGIT(*p)) {
        errno = 0;
        len = STRTOUL(p, (char**)&p, 10);
        if (errno) {
        rb_raise(rb_eRangeError, "pack length too big");
        }
    }

How do you exploit this vulnerability?

The first step in exploiting the vulnerability in a live application is having a small Proof of Concept (PoC). Let’s first try to read memory from the irb interactive shell. We will use the nice hexdump gem to display something easier to read for us humans. Without further ado here is a one-liner that will do this.

$ irb

irb(main):145:0> leak = 200; size = 2**64-1 - leak + 1 ; puts size ; "BUFF".unpack("@#{size}C#{leak+4}").map { |i| i&.chr }.join.hexdump
18446744073709551416

Offset    Hex dump                                          ASCII dump
--------  -----------------------------------------------  ------------------
00000000  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000010  00 00 00 00 00 00 00 00 65 08 93 20 00 00 00 00  |........e.. ....|
00000020  c0 7d 8c 50 97 7f 00 00 53 41 56 45 5f 48 49 53  |.}.P....SAVE_HIS|
00000030  54 4f 52 59 00 00 00 00 00 00 00 00 00 00 00 00  |TORY............|
00000040  65 40 90 00 00 00 00 00 c0 7d 8c 50 97 7f 00 00  |e@.......}.P....|
00000050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00 65 40 90 00 00 00 00 00  |........e@......|
00000070  c0 7d 8c 50 97 7f 00 00 00 00 00 00 00 00 00 00  |.}.P............|
00000080  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000090  65 40 90 00 00 00 00 00 c0 7d 8c 50 97 7f 00 00  |e@.......}.P....|
000000a0  7f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00 65 08 51 20 00 00 00 00  |........e.Q ....|
000000c0  c0 7d 8c 50 97 7f 00 00 42 55 46 46              |.}.P....BUFF|

So what are we doing here? We have a parameter for the size of the leak (leak), and we calculate the huge number that will then be decoded as a negative integer. We then create a small buffer and unpack it using a format string using these two values. We’re essentially saying: skip to this huge offset (which ends up a -leak bytes) and reads leak + 4 bytes.

This returns 204 bytes as requested.

By passing a large integer as offset, we have gone back 200 bytes before the start of BUFF in memory and then read 204 bytes (the generated format string used: @18446744073709551416C204 ).

As a sanity check, we can correctly see the content of the buffer we should be working in at the end of the ASCII dump part (BUFF). If Ruby was not vulnerable, it should never have jumped to read memory before the start of BUFF.

How do we go from the PoC to actual exploitation? We would first need to find an application running on a vulnerable Ruby (pre 2.5.1) application that we can attack. A Ruby on Rails application would be nice, as we could attack it remotely and often contains juicy secrets. This Rails application would need to have a String#unpack call were the format parameter is under attacker control. String#unpack calls are more common than you might think. They are often used to decode data coming from elsewhere (i.e. database drivers are often users of this). So to know if you are affected you might also want to take a look at all your dependencies source code…

If we had such an application, simply sending our generated malicious format string from above enables us to extract as much data as we want from the application. This allows us to read & possibly extract all the secrets that are stored in memory (database credentials, tokens) and maybe also data that would only be transiting through the application (customer credit card number or user session on a concurrent request).

Building a remediation

The easiest fix is, of course, to simply update Ruby on your machine. In the real world, this is unfortunately not always quickly doable. This reality pushed us to craft a solution that would protect all Sqreen users from CVE-2018-8778 even if they couldn’t update their Ruby version just yet.

Two primary requirements drive the development of new protections at Sqreen.

First, we can’t break our users’ application with false positives.
Second, the impact on performance should be nearly invisible.

After looking at a few options, we decided that the best solution is to “simply” hook the String#unpack method and check that the argument containing @ doesn’t include a large offset in the format string. The key here is to make sure this format string is not coming from the current request parameters.

So the rule we implemented looks a bit like:

TWO_GIGABYTES= 2**31
return false unless format_string.include?('@')
return false unless user_parameters.include?(format_string)
offset = parse(format_string)
return offset > TWO_GIGABYTES

Now let’s look at an example:

a format string C10 stops processing at the first line ⇒ no attack detected
a format string @10c12
- that is not in user parameters stops at the second line ⇒ no attack detected
- If it comes from user parameters (the code would probably be vulnerable), we check the offset size 10 and stop processing ⇒ no attack detected
With @18446744073709551416C204 as a format string, the offset 18446744073709551416 is bigger than 2**31 ⇒ an attack is detected!

And that’s it. After extensive tests, we deployed this rule to our users. They are now all protected against this buffer under-read vulnerability and can update their Ruby version when the time is right. All of this was achieved in less than 21 hours between the disclosure of the vulnerability and the full protection of our clients.

The post An in depth look at CVE-2018-8878 or why integer overflows are still a thing! appeared first on Sqreen Blog | Modern Application Security.

Context information storage for asyncio

Sqreen — Mon, 26 Feb 2018 23:35:21 +0000

At Sqreen, we are building an agent based on dynamic instrumentation. It detects security incidents from inside an application (injections, cross-site scripting etc.) and allows users to configure actions (blocking the attack, logging a stack trace etc.) without requiring code modification. The mechanisms behind dynamic instrumentation in Python are described in a previous blog post. Dynamic instrumentation is also used in Application Performance Management (APM) solutions, such as Datadog, Instana and New Relic.

Instrumenting the code allows us to execute callbacks before calling potentially hazardous functions. For example, to protect against SQL injections, we transparently wrap the method Cursor.execute with a security layer:

def sqreen_execute(self, sql_stmt, *sql_params):
    # Before executing the SQL statement, check it was not built with
    # malicious, unescaped request parameters in it.
    if has_malicious_param(sql_stmt, request.params):
        # If there is, this is an SQL injection. Abort!
        raise SQLinjection(remote_addr=request.remote_addr)
    else:
        # If not, we can safely call the original method.
        return self.execute(sql_stmt, *sql_params)

Let’s assume the instrumented code contains a vulnerable pattern like:

@app.route('/posts')
def posts(request):
    sql_stmt = 'SELECT * FROM posts WHERE id=%s' % request.params['id']

    # With dynamic instrumentation, sqreen_execute is transparently called
    # instead of cursor.execute.
    posts = cursor.execute(sql_stmt)

    return posts_template.render(posts)

Then the nominal request /posts/?id=42 will be executed (although unescaped, the request parameter is not malicious) but the malicious request /posts/?id=42 OR 1=1 won’t be. This way, we are able to protect the app without breaking it!

Context information storage

As we’ve seen above, the function sqreen_execute needs to know the current request to test the SQL statement safety. How can it get it?

Since our function transparently replaces Cursor.execute, it needs to have the same signature, hence we can’t pass the request as a parameter to sqreen_execute. Some web frameworks provide functions to get the current requests, but not all of them, and we strive for a universal solution.

What we can do is to insert a middleware (or instrument the framework’s request handling mechanism) to store the current request in a global variable:

CURRENT_REQUEST = None

def set_request(request):
    global CURRENT_REQUEST
    current_request = request

def get_request():
    return CURRENT_REQUEST

But there is a catch: web frameworks are able to handle several requests concurrently, for obvious performance reasons. So the above pattern won’t work: we may receive a first request request_1 (and store it in CURRENT_REQUEST), and before serving it receive a second request request_2 (overriding request_1 in CURRENT_REQUEST). At the time we look for SQL injections in request_1, we will mess it up with request_2! So we need a stronger, concurrent-safe mechanism to store the current request.

Thread-local storage

To tackle this issue, we need to know how concurrency is implemented. Most of Python web frameworks use threads: this is notably the case of Django, Flask and Pyramid, which are probably the most popular. They implement a common communication protocol with web servers, called WSGI (Web Server Gateway Interface) and initially described in PEP 333.

WSGI servers also use threads, along with processes, to spawn several application instances. Multiprocessing is not an issue here, since each process will handle its own copy of CURRENT_REQUEST. So, we just have to find a solution to let a service thread to store the request it is currently dealing with without impacting other threads.

And Python offers a solution for that. The function threading.local in the standard library return a namespace object whose values are thread specific. This allows us to implement thread-safe request storage as follows:

import threading

RUNTIME_STORAGE = threading.local()
RUNTIME_STORAGE.request = None

def set_request(request):
    RUNTIME_STORAGE.request = request

def get_request():
    return RUNTIME_STORAGE.request

What about asyncio?

In Python 3.4, a new concurrency model was introduced: asyncio. It provides infrastructure for single-threaded, asynchronous programming, including:

Coroutine functions, defined with async def, whose execution can be paused using the await keyword with another coroutine, and resumed once the other coroutine is completed.
An event loop to schedule and execute coroutines.

Here is an example of asynchronous code.

import asyncio

async def compute(x, y):
    print("Compute %s + %s ..." % (x, y))
    await asyncio.sleep(1.0)
    return x + y

async def print_sum(x, y):
    result = await compute(x, y)
    print("%s + %s = %s" % (x, y, result))

loop = asyncio.get_event_loop()
loop.run_until_complete(print_sum(1, 2))
loop.close()

There are two coroutine functions, print_sum and compute. At execution time

The event loop enters print_sum and immediately hands over to compute.
compute prints the computation and hands over to asyncio.sleep.
Nothing is done in the next second. If other tasks were scheduled in the event loop, they could be executed in the meantime, something that is not possible with the blocking function time.sleep.
compute is resumed and completed.
print_sum is resumed and completed.

asyncio is a great model for concurrency when IO is involved: when the code being executed is blocked waiting for an answer (for example, DB results), the program can switch to other tasks and come back to it later. It is less system-expensive than threads, and is usually faster when slow IO operations are involved.

This makes asyncio well suited for network operations and, despite being relatively young, several web frameworks have been developed around it. Among them, we recently brought support for aiohttp in our agent. This was a very interesting and challenging task since we had no support for aiohttp at all so far, and an important issue we met was with the request storage mechanism.

Here is what can happen: we receive a first request request_1 and start dealing with it in a coroutine. At some point, the coroutine is suspended and the event loop hands over to another one that handles request_2. The important point is that these two coroutines are executed in the same thread, so threading.local contains the same data for both. When the first coroutine resumes, RUNTIME_STORAGE.request has been set to request_2: that is precisely what we want to prevent.

First attempt: let’s use tasks!

What we need is a mechanism similar to threading.local that works with asyncio, i.e. lets us store context variables and keep track of values per asynchronous execution context.

Unfortunately, there is currently no built-in mechanism in Python to handle this. Different proposals have been made to provide a generic solution in future versions of Python (PEP 550, PEP 567), but in the meantime, we have to devise a solution on our own.

Let’s dig a bit further into the internals of asyncio. A coroutine whose execution is scheduled is wrapped into an asyncio.Task object, responsible for executing the coroutine object in an event loop.

Sequence diagram of the example

There is also a function asyncio.Task.current_task that returns the currently running task. Mmh… We could use this to map the current request to the task handling it. Something like this could work:

import asyncio

TASK_REQUESTS = {}

def set_request(request):
    task = asyncio.Task.current_task()
    TASK_REQUESTS[id(task)] = request

def get_request():
    task = asyncio.Task.current_task()
    return TASK_REQUESTS.get(id(task))

With this implementation, we’d also need a mechanism to ensure request deletion once the task is completed, to avoid accumulating old requests and cause a memory leak. A way to avoid dealing with it is to store the request within the Task object, as an extra attribute:

def set_request(request):
     task = asyncio.Task.current_task()
     setattr(task, 'current_request', request)

 def get_request():
     task = asyncio.Task.current_task()
     return getattr(task, 'current_request', None)

So, does it work? Let’s test!

import random

class Request:
    # Dummy request object, for the sake of testing.
    pass

async def handle_request(request):
    set_request(request)
    await asyncio.sleep(random.uniform(0, 2))
    await check_request(request)

async def check_request(request):
    # Check that the stored request corresponds to the current request. If not,
    # an AssertionError is raised and the test is interrupted with an error.
    assert get_request() is request, "wrong request"

NUM_REQUESTS = 1000

loop = asyncio.get_event_loop()
coros = [handle_request(Request()) for _ in range(NUM_REQUESTS)]
loop.run_until_complete(asyncio.gather(*coros))
loop.close()
print("Success!")

This test simulates one thousand concurrent requests. Each one is handled in a dedicated coroutine handle_request. This function stores the request, then pauses for a random duration (this simulates an async operation such as a DB access, and ensures that the coroutine execution flow is interleaved). When resumed, a nested coroutine check_request is called that ensures that get_request returns the correct request. If not, the test is interrupted by an error.

And here are some good news: the test runs smoothly with task-based request storage. It also fails with thread-local storage, which was expected but shows the test is relevant. So, have we solved our problem?

Context inheritance between tasks

Let’s try something a bit more twisted:

async def handle_request(request):
    set_request(request)
    await asyncio.gather(
        asyncio.sleep(random.uniform(0, 2)),
        check_request(request),
    )

Instead of executing asyncio.sleep and check_request sequentially, this version of handle_request runs them concurrently. This should not be a big deal: the code is a bit more concurrent, but it does not impact request handling. In particular, check_request is still called after set_request for each request.

Nevertheless, this new test fails! Something went wrong when we introduced asyncio.gather, but what?

Well, remember that scheduled coroutines are wrapped into tasks? That’s exactly what happens here: asyncio.gather creates tasks around the arguments asyncio.sleep() and check_request() and these tasks are executed by the event loop.

async def handle_request(request):
    set_request(request)                              # Running in task 1.
    await asyncio.gather(
        asyncio.sleep(random.uniform(0, 2)),          # Create child task 2.
        check_request(request),                       # Create child task 3.
    )

async def check_request(request):
    assert get_request() is request, "wrong request"  # Running in task 3.

The consequence is that set_request and get_request are called in different tasks, making the test fail. This is not due to request interleaving, as we can check by setting NUM_REQUESTS to 1: the test keeps failing.

In fact, when calling get_request from a child task, we need a mechanism to retrieve the request from the parent task if it is not defined in the child task. But asyncio does not allow us to access the parent task, so this is not going to work.

On the other hand, something asyncio let us do is to replace the function called to create new tasks, a.k.a. the task factory. This function is called in the context of the parent task, and returns a fresh child task. Well, let’s use it to decorate the child task with the current request!

Here is what a “request-aware” task factory would look like:

def request_task_factory(loop, coro):
    # This is the default way to create a child task.
    child_task = asyncio.tasks.Task(coro, loop=loop)

    # Retrieve the request from the parent task...
    parent_task = asyncio.Task.current_task(loop=loop)
    current_request = getattr(parent_task, 'current_request', None)

    # ...and store it in the child task too.
    setattr(child_task, 'current_request', current_request)

    return child_task

To install the task factory, we also need to call loop.set_task_factory(request_task_factory) before running the loop. So, here is the final version of our code:

import asyncio
import random

class Request:
    pass

def set_request(request):
    task = asyncio.Task.current_task()
    setattr(task, 'current_request', request)

def get_request():
    task = asyncio.Task.current_task()
    return getattr(task, 'current_request', None)

def request_task_factory(loop, coro):
    child_task = asyncio.tasks.Task(coro, loop=loop)
    parent_task = asyncio.Task.current_task(loop=loop)
    current_request = getattr(parent_task, 'current_request', None)
    setattr(child_task, 'current_request', current_request)
    return child_task

async def handle_request(request):
    set_request(request)
    await asyncio.gather(
        asyncio.sleep(random.uniform(0, 2)),
        check_request(request),
    )

async def check_request(request):
    assert get_request() is request

NUM_REQUESTS = 1000

loop = asyncio.get_event_loop()
loop.set_task_factory(request_task_factory)
coros = [handle_request(Request()) for _ in range(NUM_REQUESTS)]
loop.run_until_complete(asyncio.gather(*coros))
loop.close()

And it works flawlessly!

And now what?

We now have the foundations to solve the request storage issue in our agent. Since we want the agent to work as transparently as possible and not require code modification from the user, there are still two minor issues to be tackled:

We want to automatically set up the task factory. This will be done with dynamic instrumentation.
But if a custom task is set up by the user, we don’t want to overwrite it. Instead, we will wrap it up on our own.

Let’s start with the second problem. We can define a generic function wrap_request_task_factory that takes a task factory as argument and returns a variant of it that supports request propagation. The code of the wrapped function is really close to the one of request_task_factory above:

from functools import wraps

def wrap_request_task_factory(task_factory):

    @wraps(task_factory)
    def wrapped(loop, coro):
        child_task = task_factory(loop, coro)
        parent_task = asyncio.Task.current_task(loop=loop)
        current_request = getattr(parent_task, 'current_request', None)
        setattr(child_task, 'current_request', current_request)
        return child_task

    return wrapped

Then, the definition of request_task_factory can be simplified to:

@wrap_request_task_factory
def request_task_factory(loop, coro):
    asyncio.Task.current_task(loop=loop)

Time to go back to dynamic instrumentation. By hooking the import system, we can transparently replace an imported class by a custom one. So let’s define a function patch_loop_cls that creates a custom loop class with the desired behavior:

def wrap_loop_cls(loop_cls):

    class SqreenLoop(loop_cls):

        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
            # We want to use request_task_factory to be the default task
            # factory.
            super().set_task_factory(request_task_factory)

        def set_task_factory(self, task_factory):
            # If the user sets up a custom task factory, let's wrap it with
            # request propagation.
            wrapped_task_factory = wrap_request_task_factory(task_factory)
            super().set_task_factory(wrapped_task_factory)

    return SqreenLoop

This loop class transparently replaces the base one. It uses the correct task factory by default and allows the user to change it while keeping the request management layer.

Closing words

We have published most of this work (without the instrumentation part) in a Python library called AioContext. It comes with generic Context objects that behave like dictionaries. It also allows to restore the original task factory if contexts are no longer needed, and stores contexts as an extra attribute of the task factory to avoid messing with the asyncio.Task class itself. The documentation is available here.

import asyncio
import aiocontext
import random

class Request:
    pass

CONTEXT = aiocontext.Context()

async def handle_request(request):
    CONTEXT['current_request'] = request
    await asyncio.gather(
        asyncio.sleep(random.uniform(0, 2)),
        check_request(request),
    )

async def check_request(request):
    assert CONTEXT['current_request'] is request

NUM_REQUESTS = 1000

loop = asyncio.get_event_loop()
aiocontext.wrap_task_factory(loop)
CONTEXT.attach(loop)
coros = [handle_request(Request()) for _ in range(NUM_REQUESTS)]
loop.run_until_complete(asyncio.gather(*coros))
loop.close()

This work was strongly inspired by Manual Miranda’s blog post From Flask to aiohttp and the library aiotask-context. We want to thank him for the great contribution.

The post Context information storage for asyncio appeared first on Sqreen Blog | Modern Application Security and was written by Vivien Maisonneuve.