Deconstructing DevSecOps

As a an engineer that has worked in multiple fields I have seen many approaches to handle the complexity of product development. Among these, DevOps has demonstrated itself as the biggest success story to delivering software today. This success has inspired multiple offshoots attempting to capitalize on the zeitgeist of their specialized field. The advent of virtualization and cloud first gave us CloudOps and then eventually CattleOps. The ability to orchestrate containers gave us GitOps. Now AI and ML are poised to give use AIOps and MLOps respectfully. It's DevSecOps though that has succeeded where others have stumbled that we will discuss here.

Why DevOps?

It's not hard to understand why DevOps has been successful for many. There has been a collective shift of many projects to adopt agile development practices. DevOps itself is a natural progression of similar principles that extend that collaboration to the rest of the product team. Ideally DevOps would help foster a more adaptive team and product between that of the developers and that also of the operations and quality assurance (QA) team.

Breaking down the silos between these teams is a foundational concept. When the teams understand the product coherently it allows each team to solve problems more quickly. This works best when cross-team training and shared responsibilities

Another tenant of DevOps is the reliance on automation that it utilizes. This exists as either Continuous Integration (CI) and Continuous Delivery (CD) or CI/CD and Infrastructure as Code (IaC). This is only possible due to the wide selection of open source tooling that has been released in the previous decade. This has allowed CI/CD automate the build, test and deployment of products with evermore efficiency as new products and improvements are realized.

Lastly, establishing a feedback loop through testing, monitoring, and logging are vital throughout the application lifecycle. These help determine the health of the project and what action need to be performed next.

Together these establish the foundation of the DevOps toolchain. This focus on automation allows small teams do more with less and large teams to better scale.

From DevOps to DevSecOps

As a Certified Information Systems Security Professional (CISSP) for many years I have always considered security to be paramount for any project I worked. However, there was a paradigm shift occurring that I knew nothing about until I saw a meme critiquing DevOps security practices as...

Well, a mess...

It's a fact that not all projects are built the same and some are more risk adverse than others. In recent years an increasing number of supply chain attacks has only accelerated adoption of DevSecOps and put it in the forefront.

This has resulted in an increased focus to move security earlier in development. This happens to also be a core tenant of DevSecOps and is referred as a "shift-left" of security to the development phase.

Unsurprisingly, the roles of quality assurance has been replaced with an emphasis on security instead. This could be seen as a natural evolution for many projects utilizing some form of agile development due to a lesser role of requirements management in a post-waterfall world. The outcome being that lesser requirements being managed result in less requirements testing a QA engineer would perform on an application. However, this responsibility still exists but is now solely on developers in most situations.

Additionally, developers would also be responsible for ensuring that CI/CD pipelines pass various security tests:

• Secrets scanning
• Commit / Package signing
• Static Application Security Testing (SAST)
• Software Composition Analysis (SCA)
• Dynamic Application Security Testing (DAST)
• Container Scanning

An operations team would do their part by ensuring their IaC implement some type of infrastructure compliance such as a CIS benchmark and ideally some level of FIPS compliance. Since this work is usually Configuration-as-Code (CaC) and not considered code it usually doesn't have the same rigor as application code though.

All-in-all these are amazing improvements that any team should implement.

What issues does DevSecOps have?

Time and again I am reminded that there is a limit to how far collaboration can take a team. This can be because either another team has a limit to how much resources it is willing to allocate, or it is incapable of contributing regardless of its resources offered. This is often the case with cyber teams that haven't restructured or adapted the training of their personnel to support DevSecOps. To often these types are policy wonks that will happily redirect you to help desk instead of assisting anyone.

Another huge problem is with tooling ecosystem itself. While DevOps has an embarrassment of riches in open source tooling, DevSecOps instead has an endless number of licensing fees awaiting. Worse yet, many of these tools are only designed to common security issues in code. This is still better than nothing but it is pretty underwhelming when you are responsible for remediating the shear number of redundant (or duplicate) findings that have no bearing.

Once an organization begins to implement DevSecOps it can quickly spiral. This happens when the organization is unable to determine what is acceptable risk any longer. Once this happens any rapid prototyping capability will just not be allowed at this point. Suddenly, any pioneering spirit or creative capability can be strangled out of the organization.

How can DevSecOps be improved?

Cyber professionals need the correct skill sets for this type of position. Understanding software architecture is a good first step but that alone wouldn't suffice. Threat modeling is a great recommended practice for DevSecOps but useless if the team reviewing that threat model is unable to grasp it. An understanding of where vulnerabilities are and how they are exploited is much more valuable to a development and operations team then a checklist of findings.

Closing

There once was criticism about whether DevOps was strictly a methodology or a role. Today we largely associate this type of position with that of an SRE. I personally never thought this an issue, and have always considered someone who implemented the tooling and facilitated the collaboration between the development and operation teams to be just that. It is specifically the social interaction piece to break down the barriers between teams that I believe sets a DevOps engineer apart from an SRE engineer. Whereas an SRE engineer is specifically an software developer tasked with tackling infrastructure.

I think that teams would benefit more if they cross-trained. The main reason a Cyber team fails to adapt and change is that they these members or assigned these teams permanently as a scrum master or product owner are. Instead of a fully engaged participating members of a team specializing in various disciplines you are more likely to find an issue management system in place instead. Instead of engineers embracing DevSecOps as a way to interact with other an quickly resolve issues you might have to resolve yourself to get in line instead and have to prepare a documents justifying your deployment scenario.