DevSecOps Done RIGHT! Why and What to Do.

DevSecOps is something we all keep hearing, and everyone says it is important.

But why is it important? And how can we adopt DevSecOps the right way without penalizing any aspect of our workflow, especially in a Cloud Native scenario?

Let's take a look at the best practices for DevSecOps, and DOs and DONTs of this practice.

Video

As usual, if you are a visual learner, or simply prefer to watch and listen instead of reading, here you have the video with the whole explanation.

Link to the video: https://youtu.be/KaoPQLyWq_g

If you rather prefer reading, well... let's just continue :)

A Word About DevOps Transformation

In my role as DevOps Architect I get to work with the biggest, most important, and most strategic Microsoft and GitHub's clients in Asia and Oceania on their DevOps Transformations.

And you know that DevOps is a very broad term. So when I say that I work on the DevOps Transformations, the meaning of that really depends on the client.

Some companies want to focus more on technical aspects like CI/CD, branching strategies, testing, automation, etc., while others perhaps prefer taking a more theoretical approach and discuss about processes, methodologies, cultural changes, you name it.

But regardless of the approach, more and more often I work with those companies on the security aspects of DevOps, what we usually refer to as DevSecOps.

DevSecOps?

Now, even though DevSecOps is a buzzword that we hear constantly, I usually don't call it that way. The reason is simple: giving it another name makes it feel like it is something optional, something one can decide when and if to adopt. But in reality, it is not exactly like that. Let me explain.

No Security

We all know this diagram, the classic DevOps representation. If you look closer though, you will notice that there is no mention of "anything security" in it.

This of course is "no good".

Security as an Afterthought

This is why companies often operate in this way, having security completely detached from anything else, siloed, and pulled in after the fact, usually when the piece of software has already been deployed.

I hope we all agree this is still not a good practice.

Security but Detached

In fact, other companies too realize that it is not how you should operate, so they bring security closer to the other teams and practices. And they call it DevSecOps.

While this is certainly better than the previous model, I still have a problem with it. Security is still operating independently, somewhat siloed. They may have more visibility on what is happening during development, testing, deployment, etc., but the security team has yet full responsibility on anything security. And this makes it still not a very good model.

Security Everywhere

To make it better, to make our software secure and secured, we have to take this a step forward.

Security needs to be integrated in the whole flow, and practiced and applied in every single area, from ideation to coding, from testing to deployment, to monitoring.

Security everywhere and at any time: this is the key. So, how do we get there? This is where things start getting interesting, because there are multiple steps you can and eventually have to take to implement this properly.

Shift Left on Security

One of the first things people usually suggest is to shift left on security. This is a great advice, but we need to do it the right way.

This is the usual, legacy starting point in which we don't have integrated security, and as I've mentioned before the first thing companies usually do is adding security after deployment.

And this means pentesting, security scanning, bug hunting, etc. Perhaps all in production.

And don't get me wrong, those things are certainly useful. But finding a security bug in production is not a good thing, it is way too late.

The cost of fixing bugs and security issues increases every time me "move right" to the next stage in the DevOps flow, and if we are not able to catch them in time because we only apply security after deployment, a breach could be catastrophic.

For this reason we should shift left on security, starting it while in the development phase. But this is not the arrival point, this is just the beginning. As I've mentioned before, we should apply this concept everywhere.

This means having security embedded in each and every phase of our iterations. Each phase will require different approaches and tools, but we need to have it there.

To do this, it is just not possible to rely only on a central isolated security team. We need a change in the paradigm.

Security as Responsibility

Security has to become a responsibility of every person involved.

Product Owners and PMs have to think about the features with security in minds, Developers have to implement those features respecting the security indications, best practices, and guidelines, testers have to test not only for functionality but also for security... and so on so forth, you got the point.

What About Security Team?

Let's quickly address the elephant in the room. Every time I talk about this with companies and teams, I always get the same question: what about the security team? Should we get rid of it?

The answer of course is "no"!

While shifting left on security means that security is now a responsibility for everyone, that doesn't mean it's only on them. Security teams still play a very important role.

First of all, all the things like penetration testing, RBAC, etc. are still important and should still be performed.

Additionally, as I've mentioned the development team should follow guidelines, best practices, and procedures... and who's better in defining them than the security team?

Security teams have to pivot their role from being the only security-related touch point, to being a security-focused team helping and supporting all the other functions.

And as always in DevOps, with the ultimate goal of having a big cross-functional cross-discipline team rather then separated teams.

And by the way this is even more important in a Cloud-first development model, because while it is true that you delegate part of the security to the cloud provider, on the other hand your development and deployment frequency will most likely greatly increase. So you need to "do security" faster, and this model will help you scale.

DevSecOps Practice

Alright, let's talk about some more concrete practices you can adopt for securing your DevOps. (Noted as I didn't mention DevSecOps? 😉 )

Security Work in the Backlog

First thing I always recommend is treating the Security work as if it were normal work. This means including it in the Backlog, and work on it during the normal sprints or iterations.

This will help your development team especially at the beginning, because they will get more familiar with the security and technical debt remediation, and they won't have any excuse not to work on those tasks since they are part of the sprint.

Of course, if there is any security incident or bug that needs to be remediated immediately, that will have an exception. But in general, this should be the rule.

Security Throughout the Process

And once you have it in your work management software as backlog, then you should carry it through your whole process.

• Every time you open a Pull Request, there should be a security scan running for it and it should block the PR if vulnerabilities are present. This way, you stop vulnerable code to reach your default branch.
• Then, have the same scanning in your main CI, meaning the CI on your default branch, for each push or merge.
• Furthermore, schedule a scan at a fixed interval, perhaps weekly, so you can make sure that your codebase is constantly checked even if you do not commit or merge anything, should new vulnerabilities be discovered in the mean time.
• Finally, deploy only if all the security tests have passed, and deploy across multiple stages with gates in place to further test for security and vulnerabilities, aside from proper functionality.

Enabling progressive exposure like I've just described will allow you to identify and fix security vulnerabilities as quickly as possible, and even roll back or roll forward much more easily if you have to.

Assume Breach

And speaking of identifying vulnerabilities, there is one mindset change that can help you with that: assuming breaches. There are different ways to do it, but today I want to tell you how we've addressed that in the Microsoft Engineering group working on Azure DevOps.

As many companies, we started with classic Red/Blue war games to the learn attacks and practice responses.

But over time we've ended up eliminating the Blue team and shifting even this part left. Which means that now is the engineering team that directly that takes care of it.

And this approach has helped us preventing some of the most common issues like vulnerabilities in Open Source dependencies, secret leaks, credential theft, and other.

DevSecOps Everywhere

Let's now quickly take a look back at the general definition of DevOps. The definition that in my opinion best represent DevOps is this one from Donovan Brown:

DevOps is the union of people, processes, and products to enable continuous delivery of value to your end users.

People, Processes and Products are the pillars of DevOps. And so, to have the maximum benefits and results out of DevSecOps, we need to find and apply techniques to each of those 3 pillars.

For once, let's start with the products, which can also be referred to as technologies or tools. There are many things that you can do on this pillar, and some are very dependent on your specific technological landscape. But others are universal.

For example, automating as much as possible will take the human error out of the equation. Also, using Infrastructure as code and Config as Code will ensure you reproducibility and security in the configuration as well. Of course, the usage of tools like Static Code Analysis and Application Security Testing, both Static and Dynamic, as soon as possible in the development cycle helps in identifying issues as quickly as possible. And this is especially true if you run those tests in the Pull Request phase. And same considerations apply for other scan types like Credential Scanning or scanning for known vulnerabilities, either in code patterns or dependencies.

But tools alone are not enough, you need to have Processes in place to support this. And those processes can be complex like Secure Development Lifecycle, Thread Modeling and the Red/Blue team exercises, just to name a few, or simpler things like adopting code reviews across the whole organization (of course in this case with emphasis on the security of the code), limiting access to production (which should be virtually non-existing for non-automated tools), adopting progressive exposure as I've mentioned before, and of course doing regular security assessments.

But once again this is not enough. You can have the best tools in the market, and the best processes defined... but if your people won't use those tools or follow those processes, you won't be successful.

And for the People, all starts with the education: to let the teams embrace a security first mindset, to always assume breach and try to prevent it, and even for more basic things like protecting credentials, whether we are talking about some external service or environment credentials.

People are, as always in DevOps, the most important part of the equation. So remember to invest in your people, in your teams, in your company culture.

Selecting DevSecOps Tools

People are important also when it comes to the selection of the tools. It is incredibly important to find the right tools that fit your processes and that your people would actually want to use.

You need for example to find tools that are Developer friendly, and that means easy to use, fast, and reliable. If something requires a lot of effort or time to be used or run, then nobody will use it. And if it takes too long to execute, then your builds and CIs will slow down, impacting your cadence.

Another thing you should look into is having tools that can be used both locally and in a CI system. This way, the developers can run those tools on their machines, since they are fast (as we said before), before even committing the code to the remote branches and opening their Pull Requests

And since we are talking about security tools, the tools you use must have a very low false positive rate. You don't want to waste your time investigating hundreds of false positives. Plus, if you have real issues or vulnerabilities they would be buried under many non-issues and that may make the whole process flawed.

Cloud Native and Containers

If you develop in containers, remember to have in place both application scanning and container image scanning. This way you can ensure that there are no vulnerabilities in your hidden dependencies, meaning the dependencies that you inherit from the Container Base image itself, and that all the policies are respected.

If the container registry you use has an embedded security scanning capability, it is usually a good starting point. But you would still want to have your images scanned as early as possible in your chain.

And still talking about containers, there is something which is very often overlooked: I'm talking about Kubernetes best practices validation and enforcement.

There are just a few tools out there to do this (I personally usually use Datree, as I explain in this video here) , but in my opinion it is an extremely important area.

The whole concept is to prevent security issues in production by preventing Kubernetes misconfigurations from reaching production. And of course you can do this manually via Code Review, for example, but code reviews are time consuming and, most importantly, they are error prone. If you can automate things like Kubernetes schema validation, and the enforcement of best practices and policies over your Kubernetes manifests, then you will make sure that you can just focus on your application bits.

And, once again, ideally you should find a tool that does this kind of validations in every scenario, including local development environments, and that does it very quickly and reliably to maximize the benefits. (and this is one of the reasons why I use Datree 😉 )

Recap

Alright, let's wrap this up and let's just recap what we've seen:

• Security is not just for security people, or security teams. It should be everyone's responsibility.
• Apply security everywhere, in every step of your software production flow, and as early and as frequently as possible. This will maximize your chances to identify and solve security issues fast.
• Consider the 3 pillars of DevOps, People, Processes, and Products, and identify areas of improvement in each of them when it comes to security. They are all important, and you need to have solutions for each and every of them.
• Choose the tools that people would want to actually use: fast, easy to use, and reliable.

And finally, DevSecOps should just be DevOps. Security should always be embedded in DevOps... so, make it so and you'll soon see the benefits.

Conclusions

I hope you've found this valuable. Please feel free to reach out to me if you have any questions, just comment down below or check my contacts in the next section.

Also, let me know what you think about my guidelines for an efficient DevSecOps adoption.

Like, share and follow me 🚀 for more content:

📽 YouTube
☕ Buy me a coffee
💖 Patreon
📧 Newsletter
🌐 CoderDave.io Website
👕 Merch
👦🏻 Facebook page
🐱‍💻 GitHub
👲🏻 Twitter
👴🏻 LinkedIn
🔉 Podcast

Comments

[Bless Darah Gah]

This was quite an extensive and good read. As someone who is new to DevSecOps with as software engineering position, it makes perfects sense that we shift everything to the left and consider security from the get go.
Awesome!

[Krav]

*A+ for effort & a great read. *