Firstly, let's start with 2 commonly heard statements:
Enterprise doesn't do DevOps
or
Releasing is hard
- signed, every enterprise developer ever
Whilst Enterprise companies tend to have a lot of bureaucracy (especially around releasing), we should still look to improve the pace at which we deliver working software to our customers.
Does this apply to my Enterprise org?
Yep! Any organisation whose customers, regulators, and developers appreciate faster, more reliable software releases with less risk can employ these thought processes.
What is DevOps?
Let's start with a simple shared vision of what DevOps actually is.
DevOps is
a set of practices
that works to
automate processes
so software teams can
build, test, and release
software faster and more reliably.
From Atlassian DevOps
The target
What are we looking to achieve here?
We are hoping to:
- Reduce the time it takes for our work (including bug fixes) to land in our user's hands
- Reduce the time it takes to become aware of a problem in Production
- Reduce the risk and longevity of downtime due to a release
- Reduce the impact of any particular bug introduced
Ok, but how?
Git master is sacred
You must ensure that the latest version of your code is always deployable. If you need to "cut a release" and branch off master, you will be less inclined to release on a regular basis.
If your code is not "ready to release" at any time, your ability to safely respond when shit hits the fan is heavily affected.
Think about it, if you first need to rollback your code to a "known" safe place, then this is yet another step in the process before you can even begin fixing your production incident.
In order to keep master "production-ready", all merges to master must also be ready for production.
This is achieved by 2 key things:
- Automation
- Anything that gives you the confidence to release, should be automated and it should prevent branches from merging to master if they fail
- Your automation should make it VERY hard to accidentally break production
- Coding Practices
- Write your code in a way that allows you to deploy to production safely without causing an outage due to a mistake.
- Feature Toggles, Keystones, and backward compatibility are just some of the techniques that can help maintain prod readiness.
Part 2 of this series "Coding Practices" will give some examples of ways to ensure your code is production-ready.
You build it, you run it
The person best placed to fix issues with a feature is the person that just wrote code for it.
If that person is in charge of dealing with any customer issues with that piece of work they will:
- Better understand the problems when they occur
- Be more inclined to take care when writing the code
- Have better context and incentive to improve the supportability of that code in the future
By getting your Devs at least involved in this process, your software quality will be given a good chance to improve, if only through empathy for the user and themselves.
See Atlassian's take on You build it, you run it and another on some of their ideas around site reliability engineering
Small releases are best
If you spend a year developing software before giving it to a user, you are spending a year building up risk. Any piece of that years (or even months) development could (and likely does) have an undiscovered bug.
If anything goes wrong (and it will), you will have to sift through a whole year of code and features to discover the issue.
On the other hand, if you make a single change and deploy it to production you will likely have much more confidence about deploying to production. Why? Well, if it breaks, it is much easier to have confidence around which change caused the issue... And you can simply rollback to the previous version (or even write a test, fix the bug, and redeploy).
Due to our new confidence in finding issues and rolling back changes quickly, the risk of any 1 change is drastically reduced. We also massively increase our ability to avoid downtime due to an introduced bug.
With additional confidence, we can release faster which gives us much better context when things do go wrong since, we have only just finished working on that section of the code.
This only helps give us the confidence we need to release at an even faster cadence and so on.
Simply stated:
Small change == less risk == faster to release
For further reading here is an interesting Kent Beck article about taking small changes to the extreme: Testing the boundaries of collaboration
Here is also an interesting read on the usefulness of pull-requests, I think the key takeaway here should be that small, very short-lived branches are best.
Automated testing is the key
The other lynchpin in our DevOps plan is testing. If we don't have sufficient automated testing to give us the confidence to release then the plan falls apart. If we rely on manual testing for this confidence, and this manual testing is performed AFTER merging to master, then your manual testing becomes a blocker to production. However, because you have merged to master, you don't just block your code from being released but also any other code that is merged after yours. This puts pressure on the QAs, causing unnecessary delays or forcing them to drop their testing standards.
This is not to say that manual exploratory testing doesn't have its place, it can.
Just that with sufficient automated testing, manual testing shouldn't block a release. If used correctly however, it can be used to validate functionality (often best done by someone with first-hand experience in the business). Manual testing should never be used for anything that you ever want to run more than once to give you confidence you haven't broken something.
For 2 very interesting points of view on appropriate automated testing levels, check out these articles by:
Kent Beck
Kent C. Dodds.
(Apparently, all Kents have strong feelings about tests!)
However, if your QAs are finding issues that if deployed to production would impact users, then you are in luck! The next post in this series "DevOps Coding Practices" will take you through several techniques to avoid blocking production deployments, whilst still maintaining safe and reliable production environments.
Let's put it into practice
Next up are some simple DevOps coding practices to follow to ensure master is ready to release at any time.
Top comments (3)
I’m at the beginning of my DevOps journey and all I can say is that I’m excited to learn more about it and face its challenges.
Thanks for sharing!
Great to hear. Care to share your journey so far? Or how you are planning to go about it?
Sure!
I could say that this journey has been a bit scattered. I heard about Docker more than one year ago, but I didn't set up to properly learn and build apps with it ever since. I just learned it passively, aiming to only understand the concept. Right now, I could say that I'm quite familiar with it, especially because I got some opportunities to get my hands dirty.
I've also been very curious of how such tools work under the hood and this was also a great chance to learn Go. I believe this also helps me improve my Linux knowledge(which I heard is kind of necessary in DevOps space).
Recently I've started dabbling a bit more with docker-compose, while keeping an eye out for other interesting tools like k8s, Terraform, Ansible, Kibana, Prometheus, which I have almost no idea what they're used for, but I can't wait to see where they fit. I also heard that being acquainted with a cloud provider is also something good to have. I'm also curious about metrics, which at the moment I know nothing about.
Regarding my next steps, I'm trying to take on DevOps-related tasks at my job too, while learning about practices and tools in my free time(which is kind of hard to find while studying in college :) ).
I find this space very interesting and I think it contributes to my goal of becoming the best developer I can humanly be.