Separating Release from Deployment

Release and deployment are important parts of the software development life cycle (SDLC). In fact they're critical because they are responsible for how a piece of software is delivered to its end users. These two steps in the SDLC are often treated as one, coupling together release and deployment. However, just because a code change has been deployed, does not mean it has been released to customers, or is ready to be released to customers.

When working with a DevOps mindset, there is no separate team managing release and deployment. The development team is responsible for their application throughout the entire SDLC - including deployment and release. Many modern teams have automated pipelines that trigger after merging changes into their main code branch. These pipelines deploy their code changes to any number of pre-production and production environments automatically, running tests and checks as they go. When the deployment is successfully completed to production, the changes are immediately available to release to customers. In most cases, they are immediately released to customers without further thought.

What about big features?

One of the main tenets of DevOps is to deliver value to customers quickly and iteratively with shortened feedback loops, as opposed to having long release cycles. However, big features still exist and long-lived development branches are awfully big pains to maintain, code review, and test. The "big bang" style of deploying these large changes also leads to a lot of developer pain, bugs, and incidents.

If deployment and release were separated from each other, the need for big, long-lived development branches goes away. Code changes could be done in smaller, iterative pieces that build on each other. These smaller pieces are easier to test, code review, and deploy. So how can we deploy changes iteratively without releasing?

Decoupling deployment

A simple option to separate deployment from release is using feature flags to turn features or code blocks on and off. When code changes that are not ready for immediate release to customers are merged into main, they can be deployed behind feature flags which would determine whether or not that feature was visible in a given environment. The feature flag could be turned on in test environments to use as the rest of the feature is developed, while it could remain off in production to completely hide the feature from customers and continue running the current software.

More complex solutions, such as traffic shifting, can also help to decouple release from deployment. With traffic shifting, a small percentage of traffic is directed to the new code path while the majority remains on the current code path. As the new code path proves to be effective, the proportion of traffic sent to it is increased until all traffic is directed through the new code path and the old one is unused. This can be automated using a variety of tools, or even set up as part of the infrastructure as code (IaC) configuration for many resources, like AWS Lambda.

There are many other approaches a team could take to decouple deploying software from releasing it, and each team will need to find what fits their processes, tooling, and team the best. However, decoupling deployment from release should not introduce manual toil. Wherever possible, codify and automate these processes!

Code Freezes

In the past, companies have implemented code freezes to prevent introducing changes to the codebase for a period of time. This could be before a big release so that the quality assurance (QA) team could perform tests, during holidays when there was little support, or times of the year when their business is sensitive to change (for example, Black Friday for an American e-commerce company).

It's simple to say that a code freeze would go against the very concept of continuous integration and continuous delivery (CI/CD) and the DevOps methodology of continuously delivering customer value. However, it is also simple to understand that deploying code or releasing features without adequate support available is irresponsible. When operating with a DevOps mindset and using automated pipelines that deploy changes as soon as they're merged to main, it is easy to forget that the code is also being released to customers. In this case, a code freeze means preventing teams from merging any code to main and essentially piling up changes to be delivered later on in quick succession, or even using a dreaded long-lived development branch. This brings with it the same issues that "big bang" deployments have.

It also brings in a new set of problems. Top-down organization-wide decisions that impact how a team manages their application, like implementing an organization-wide change freeze, undermine the very essence of autonomy and trust. If teams are operating with a DevOps mindset and a company is truly embracing a culture of team autonomy, then they need to enable teams to make the good decisions for the things that they own, and support them when mistakes are made.

Responsible Application Ownership

Instead of implementing an organization-wide code freeze, why not give teams the autonomy, tools, and knowledge to be responsible application owners?

A responsible application owner considers the implications of their actions, and is able to determine the impact of deploying or releasing a code change. They should be able to identify and understand the potential risks associated with a change to their application, and how to mitigate them. Teams should always take into consideration who is available to monitor a deployment or release when either action is taken and that includes knowing who is available to support customers. For example, instead of merging a code change at 5pm on a Friday when they're about to take off for the weekend a responsible team might decide to wait to do it Monday morning when they will be there to deal with any fall out from the deployment or release of that change, and customer support engineers and the SRE team will be around to assist should something go wrong.

With ownership comes responsibility. It shouldn't matter what time of the year it is, teams should be empowered to own their application's software development life cycle end to end - and that includes making conscious decisions about deployment and release.

Conclusion

When it comes to deployment and release of an application remember that they are in fact two distinct phases of the software development life cycle. Empowering development teams to own their applications throughout the entire SDLC means enabling them to make conscious choices when it comes to deploying and releasing code changes.

Next time you're about to merge a code change, pause and ask yourself:

• "What could happen when this is merged?"
• "What is the impact if this is released to customers immediately?"
• "Who will support the application if something goes wrong during deployment?"
• "How will the team know if this was deployed successfully?"
• "How will the team know if this was released successfully?"

Take responsibility, and make conscious decisions when it comes to deploying a code change and when releasing it to your customers.