A detailed look at what DevOps really is with the goal to clarify all questions and doubts around it β
This is the written version of my new youtube video βοΈ π
DevOps has been gaining a lot of popularity and is taking over the traditional way of software development.
Difficulty to define DevOps
However, "DevOps" term itself is so broad and includes so many things that it became difficult to exactly define it and clearly set the boundaries of DevOps compared to other IT fields. So this post tries to answer the question of "what DevOps is" in detail.
The simplest definition is that "DevOps is an intersection of Development and Operations".
But where do boundaries of DevOps start and end? Which part of development is not DevOps? Or what part of operations is not DevOps? And why was there even a need for something between development and operations? π
Development and Operations are two main components in the whole application release process:
So let's look in detail at this release process starting from the very beginning!
Application Release Process
Whenever we're developing an application, we always have the same process of delivering that application to the end users. So this is the main goal no matter if you use waterfall or agile or whatever approach. At its core: 1) you create an application 2) and you want to deliver it to your end users so that they can use it. π©βπ»
So let's say you have a great idea about a cool application. You define its functionality or in other words what features it will have,
- you code it,
- you test it
- and now that you have a tested application you want to actually deploy it on a public server and let users access it.
For that you build and package your application in some kind of executable form so that it can run. You configure the public server with all the needed stuff like installing any tools the application needs and deploy your application there, you configure firewall rules to allow access to the application on the server and you have launched, users can start using it! π
So that's the simplified basis of any application release, but that's not the end of the journey. While in use, you of course have to check in on your application:
- Is everything running fine?
- Are users experiencing any issues?
- Maybe there are bugs in the application that you didn't catch when testing
- Can application handle high user loads?
- etc
So after launching it, you have to actually make sure that your application is accessible and usable by end users and if there are any issues for users, of course you should fix them.
Now that was the initial launch of your application, but the application development is not done yet. If you see users like your application you would want to make it even cooler, add new features, maybe optimize the performance by getting better servers or making your application faster and so on. So you still have a lot of things to do and every time you improve your application either the code itself or the server configuration you want to make this improvement accessible to the end users immediately.
So after the initial launch you do multiple updates to your application and to keep track of these updates you version those changes.
And you do that over and over again:
- you have an idea of improvement
- you implement it in code
- you test it
- build and package it
- you deploy it
- and once released, you observe it in the production to see whether there are any new improvement possibilities or any issues that need to be fixed right away
So this gives you a process of continuous delivery of changes an endless cycle of improvements to your application.
And DevOps is about making this process of continuous delivery fast and with minimal errors and bugs:
So with DevOps improvements get created and delivered to users fast, but also those improvements are of high quality and well tested. And that is a big challenge: quickly delivering high quality code. π³
Challenges DevOps tries to solve
Now let's see what are exactly the challenges that teams may face during this process and which DevOps tries to solve.
During this whole release process we have roadblocks and frictions:
Now what are the frictions and roadblocks in the release process? π€
1) Miscommunication and Lack of Collaboration βοΈ
First and the most important challenge is miscommunication and lack of collaboration between developers and operations.
So releasing application has two main parts:
- you code the application
- you deploy and run the application
Developers are responsible for coding. Operations are responsible for running the application.
And between these two there might be a gap of: "I wrote an application, but I can't run it" or "I'm running the application, but i don't know how it works" ππ»ββοΈ:
So developers would code without considering where or how the code will be deployed, while operations would try to deploy without really understanding what and why they are deploying or how the application even works. And this would result in miscommunications between these two.
For example:
Developers finish coding but the deployment guide for the operations team is not good enough or well-documented enough, so operations team struggles deploying it, so release takes longer.
This kind of miscommunication could cause stretching the release periods for days and weeks and in complex badly maintained projects maybe even month.
So between the developer is done with the feature and operation starts deploying it, there is no clearly defined automated process of handover. It's based on a complex bureaucratic process of what checklists π need to be completed and what needs to be documented and who needs to manually approve what for the release and so on. So no streamlines or automated processes here.
2) Conflict of Interest βοΈ
Apart from miscommunications between development and operations, in a traditional setup, where one team is only responsible for development and other team only for operations, these two have seemingly different incentives that make it hard for them to work together.
Developers want to push out new features fast, while operations want to make sure those changes don't break anything, because operations are incentivized to maintain stability in production. Their main focus is to make sure the application is available, doesn't crash, doesn't show 500 errors to the users and so on.
This means that operations need to resist the speed of release and check all the aspects of a new release to make sure it's 100% safe, which again slows down the process, especially considering that operations don't really understand the code or the application. So it's even more effort for them to evaluate this new release.
So even though the main common goal of everyone in a company should be to deliver high quality applications to the end users fast, in practice the more immediate goals are for each role to do their job. And developers job is to quickly create new features and push them out, operations job is to maintain the system stability and resist new changes being pushed out:
This gives us a conflict of interest and this kind of setup naturally makes it difficult for those two to collaborate. π€·ββοΈ
3) Security βοΈ
Another showstopper when releasing a feature is Security. Just like operations team carefully evaluates and changes to make sure they won't affect system stability, security team will evaluate any changes to make sure they don't affect systems security:
And in a traditional setup this is the same manual bureaucratic process as operations, which takes days or weeks and slows down the release process.
And as I mentioned DevOps is about removing any roadblocks that slow down the process, so it includes this one as well.
However, even though this is part of the DevOps solution, there was a separate term created for that called "DevSecOps" in order to highlight and just remind the teams of the importance of security, because it somehow got left out.
I actually have a separate dedicated video on DevSecOps, which you can also check out if you're interested: DevSecOps explained
4) Application Testing βοΈ
Now adding to the list of show stoppers is application testing. In many projects there are separate teams or roles for testers, who test the application changes on different levels:
- like testing just the feature
- testing the whole application,
- testing on multiple environments etc.
And often these tests are done manually, when teams cannot totally rely only on their automated tests and only after manual testing is done, can the change be released:
Even though this may not be done by development or operations role, but rather a separate tester role, this is an important part of the release process and may also slow it down considerably!
5) Manual Work βοΈ
As I mentioned, many of the tasks during the release process like testing, security checks, deployment etc used to be done manually. ππ»ββοΈ
For example, operations would do most of the operations tasks manually, either by directly executing commands on the servers to install tools, configure stuff, do patches or have scripts or small programs they execute. But in both cases this is manual work:
Drawbacks of manual work
This manual work is slow and more error prone, because of human error plus with manual work you have a disadvantage that knowledge sharing is very difficult, because people who do the tasks would have to document it and others would have to read it.
It's also very intransparent, because it's hard to trace, who executed what when and finally when infrastructure configuration and so on was done manually, if something happens to the infrastructure, it may be really hard to recover and replicate the exact state fast. You would have to remember exactly what was done on the servers, in which order to get to that previous infrastructure state. π€¦π½ββοΈ
DevOps tries to remove these roadblocks β
So you see the main characteristic of all these issues is that they all slow down the release cycle and create roadblocks on the way. And you also see in case of security and testing, that DevOps may even go over only development or only operations responsibilities and tasks:
That's why to understand DevOps instead of focusing on the name and what it means, we're focusing on what it tries to achieve: π‘
DevOps tries to remove all these roadblocks and things that slow down the release process, whatever that may be and instead of manual inefficient processes helps create fully automated streamlined processes for release cycles.
And this can be done step by step, removing one roadblock at a time until you have a fully optimized and automated DevOps process, that makes your application release super easy. π
DevOps Concept as a Solution
So how does DevOps help achieve this and solve all these challenges? π
Well, by the official definition and this was the original idea of DevOps:
DevOps defines a combination of cultural philosophies, practices and tools for doing that.
So DevOps is not just one set of tools or one specific concept, it's a combination of anything that creates the process of releasing the software fast and with high quality. And the main part of the concept was that developers and operations people should work together more often, talk to each other more often and collaborate better to achieve that.
DevOps in Practice: DevOps as a separate Role
But actually this definition is too broad and too high level and makes it hard to imagine how it works in practice. π So it's just not specific enough. So naturally different companies implemented DevOps in different ways. So the actual implementation of DevOps looked pretty different from company to company.
But since companies started adopting it, gradually it got a more concrete form with some of the common patterns across many companies and one of these patterns was that DevOps evolved into an actual role called a "Devops Engineer", where either developers are doing DevOps as a job next to development or operations are doing it or someone is doing DevOps exclusively as their only job.
And a set of technologies that were used to implement the DevOps principles became DevOps technologies, which now DevOps engineers would need to learn:
And I understand that many people are resisting the idea of DevOps engineer and the creators of the DevOps concept didn't see it used this way, but the reality is often different from the theory. We see that concept was adjusted and bent to meet the needs of the end goal and DevOps engineer role is what came out of it. π€·ββοΈ
And that DevOps role is responsible for creating a streamlined release process without any roadblocks slowing down the release and that's why in the center of DevOps is the well-known Continuous Integration / Continuous Delivery process
DevOps in Practice: How to become a DevOps Engineer
Check out my other blog article to see:
After reading this, you may be thinking, these are a lot of things to learn and it may be hard to know, where to start or what to learn first or what resources to use. π€―
Well, there are many resources out there to learn individual DevOps technologies, π I actually cover many of the DevOps technologies on my Youtube channel. π
But ideally you want to follow a well-structured step-by-step roadmap and more importantly learn how to use these technologies together in combination, because that's what DevOps engineers do. They use and integrate multiple technologies together to create DevOps processes and of course you want to learn all these with actual real-life project examples to know how it will look like in a real job. And very few courses and learning resources offer this and that's exactly why we created a complete DevOps Bootcamp with a clear structure and lots of hands-on projects.
So if you're thinking about becoming a DevOps engineer or slowly transitioning into DevOps you can definitely check out our DevOps bootcamp π
DevOps vs SRE - How SRE fits into the whole DevOps process
To get a full picture of DevOps I want to mention one more concept, which is SRE or Site Reliability Engineering and how it fits into DevOps.
We just learned that there are two definitions of DevOps
- the original definition, which is more high level and more broad and doesn't specify how exactly DevOps should be implemented
- and a more practical one, which evolved over time with its own DevOps engineer role.
So when we compare DevOps with SRE it's important to know which definition of DevOps we're using for this comparison?
1. First broader definition of DevOps vs. SRE
DevOps is a more high-level concept that defines what needs to be done to achieve the automated streamlined release process, while SRE is more specific about how to exactly implement this process and how to implement DevOps principles.
So many people would say that SRE is a specific implementation of the DevOps concepts:
2. Practical DevOps vs. SRE
But as we saw DevOps itself also became more practical with its own role and specific technologies and ways to implement it. So what's the comparison here? π€
Well, in many companies this practical DevOps implementation became more focused and concentrated on the speed of delivery for the application changes and of course even though it's part of the DevOps principles to not only release fast, but release quality code many DevOps teams in practice again seemed to optimize more for the speed than reliability.
So as a great complementary part of DevOps, SRE emerged with the same principles and goals in mind, which is release quality code fast, but as the name suggests more focused on reliability and keeping systems stable, while allowing for fast changes:
So SRE is its own role with its own set of tools for making systems reliable. So these two were kind of parallel developments and are now often seen as two sides of the same coin and it's not uncommon for teams to have both DevOps engineer and SRE helping implement the DevOps principles.
Own 'What is SRE' video coming soon.. π¬
This was just a short look at SRE to understand it in comparison to DevOps, but since I have received many questions about what SRE is, I will release its own follow-up video on SRE in the next weeks to explain in more detail like how SRE works in practice, what are the tasks and responsibilities of a Site Reliability engineer and so on.
For DevOps I hope I could clarify all your questions about it. If not leave a comment in the video and I will try to answer them π
Good luck on your DevOps journey! π πͺ
Like, share and follow me π for more content:
Top comments (3)
Watched the video on youtube and it was awesome! Thank you.
I'm trying to learn DevOps. I have started with Docker and Nginx. your Youtube channel is the best reference for DevOps.
Looking forward for the next videos.
Great article to describe what is DevOps.
It would be interesting to explain concretly what it means for each persona. Because everybody is worried of change, so when we start using new words, one might be afraid.
The automation and streamlined processes are key to achieving DevOps. It might means:
Some companies keep separate roles (Dev vs Ops), but try to make them work smoothly, openly and with a clear and honest communication, in a friendly and working environment. Some others dissolve both roles into "DevOps" and everybody experience increase as anybody should be able to help on any step of the release process.
Which do you prefer?
I believe both roles skill sets overlap somewhat and it's nice knowing some operations from developers lense. But when it becomes serious business, large scales, both roles require different expertise that I don't think many people could be highly specialized in both domains.