loading...
Cover image for Dirty secrets of DevOps

Dirty secrets of DevOps

liquid_chickens profile image Chris Dodds ・4 min read

I've read dozens of DevOps success stories, tales of bold IT leaders transforming their business and steering big corporate ships into the future. It's hard to avoid all these stories about "DevOps transformation" and how some guy named Jim pulled the IT department out of the stone age. The trades, blogs, and conference presentations are filled with them.

No one talks about the failures though, very few even write about their struggles implementing DevOps. It's all sunshine and rainbows, which sucks, because that isn't real.

Success teaches us little other than survivorship bias and how to feel bad about what we haven't achieved. Failure and hardship are where we learn. That's where the good, meaty stuff is.

So here are a few dirty secrets of DevOps.

Most companies that say they are "doing DevOps", aren't.

Because of all the success stories (real or imagined) that have wiggled their way into the minds of CXOs, "we should be doing DevOps" became an empty corporate directive that inspired thousands of executives to start calling their IT infrastructure groups "DevOps" instead of "Infrastructure" or "Systems" or "Operations".

Unfortunately, this seems to often play out as "We've renamed the group, so we're going to be letting most of the team go, because we're DevOpsing all the things now and being lean and mean. Also, the developers are still a separate group and they'll be throwing more stuff over the wall to you."

So you end up with a few overworked, traditional Ops folks trying to keep the wheels on the bus with zero changes to the way work is managed or how the IT group functions day-to-day. Their manager is shouting down the hall about automation while the poor Ops team is trying to pivot a SAN-installing, server-racking skill set into something that looks like a cave-man version of coding.

The only metric that improves in this scenario is "personnel cost", and only temporarily because burnout and churn spike, driving up staffing costs a few quarters down the road. But it looks good long enough for someone to say "See, we did it!" and feel validated.

Even if you get the "IT folks" on board, getting plugged into the business so DevOps practices can benefit other groups and the overall bottom line comes with its own challenges.

Fixing this issue requires a lot of skill managing upward and sideways. Often times, it's not worth trying to change and moving on is a better option. Your mileage may vary.

Implementing DevOps is Really Fucking Hard™

DevOps is all about people and process, getting everyone working together to do fewer dumb things, and smart things faster.

Historically, getting people to work together and not be jerks to one-another has been a bit of a challenge. Humans achieve awesome things when we collaborate (like spaceships and lasers), but we usually suck at working together. Because of that, I'm always impressed when I come across an excellent people-whisperer, someone who can motivate different groups to work towards a common goal without burning down each other's village.

Problem is, there's like five of those people on the face of the planet and chances are, they don't work for your company. You might have lucked out and have 1 or 2 folks who are kind of OK at people-wrangling and peace-keeping, but most businesses (especially the bigger ones) seem about three seconds and a passive-aggressive sticky note in the office kitchen away from an all out blood bath.

Assuming you can get people working together, you're now faced with the challenge of implementing process. You probably have one person on your team who loves process. Everyone else hates process and that person.

You're never finished

There's no such thing as "we achieved DevOps". It's a practice like healthy living or Buddhism. There has to be a champion(s) on your team who pushes every day to make things better.

When someone talks about DevOps success, what they're really talking about is achieving flow, that there is a functional work process in place that is continually measured and improved upon. It's an ouroboros value pipeline.

That's not something you can arrive at and stop tending to. Without constant care and feeding, the processes you worked so hard to implement will start to die off.

No champion, no DevOps.

All that being said...

None of this means that DevOps isn't worth doing, just that you need to be realistic about the challenges you're going to face. I've leaned on hyperbole pretty hard to swing the pendulum away from sunshine and rainbows, because reality is somewhere in the middle.

Getting Dev and Ops (and Security), groups who have traditionally walked down the hall waving their middle fingers at each other, to 1.) work together and 2.) implement and adhere to process, will likely be one of the most difficult things you've attempted in your career. You have to put a lot of work into making the right things the easy things, reducing friction wherever you can. Setting mandates or badgering doesn't work, you have to sell the value.

Getting top-down buy-in (and understanding!) of true DevOps/Agile practices is hard. It requires reorganizing groups and a sustained sales pitch to all involved. The need for this trails off a little once the business and IT staff start seeing value, but expect it to be a long, sustained effort. I'm always a bit dubious when I hear something like "we transformed IT in three months" - either that group really has their shit together, someone is lying, or we're not using the same dictionary.

For practitioners and evangelists, these are the things we need to start talking about more. There's a slick consultant vibe that's weaved throughout discussions about DevOps that glosses over the practical and prescriptive. Too many of these conversations focus on high-level what-to-dos and not enough on concrete how-tos and context, especially when it comes to people-centric issues.

Originally posted on chrisdodds.net

Posted on by:

liquid_chickens profile

Chris Dodds

@liquid_chickens

My background is in traditional enterprise infrastructure, but the last few years I've focused on cloud and automation via DevOps.

Discussion

markdown guide
 

Isn't devops mostly about writing Chef/Puppet scripts and setting up AWS infrastructure? It'd be helpful to know about the field for those of us not in it

 

DevOps is about :

  1. Culture (Collaboration, Communication, Organisation, ...)
  2. Processes (Continuous Delivery)
  3. Toolset (Configurations Management, Cloud computing, Container technologies, ...). And here's where you have Ansible, Terraform, AWS, Docker, Vagrant, ...
 

We're not big enough to bother distinguishing DevOpsy work too much from non-DevOpsy stuff, so we don't really think in some of these terms, but this stands out:

You're never finished

There is definitely an urge at every step in the process to feel like "Okay, once we get these few things in place, everything will be way smoother and automated and we can just focus on building features" but that day isn't going to come. It's not even going to come in the sense of "okay this portion of the team maintains the processes and the other folks are free to proceed uninhibited."

It's going to be the case that we are always going to be refining and evolving these things, and hopefully preemptively because we don't want to get caught off guard by a big shift.

 

I've run across a few really smart managers who build refactoring time into all their sprints that covers both core code and automation exactly for those reasons.

 

Thanks for your post.

I was really curious what would come, when I started reading your post, since you started your post like you wanted to say that and how DevOps sucks, while you actually said it's hard to implement and it's not an end in itself.

I'm not convinced that the whole Internet is "all sunshine and rainbows" when it comes to DevOps, but I can totally relate to the point you actually made.

As a matter of fact DevOps is about collaboration and collaboration is hard. Full stop.

Best Regards,
Patrick

 

Excellent post Chris - some genuine LOL lines, especially this one:

...most businesses (especially the bigger ones) seem about three seconds and a passive-aggressive sticky note in the office kitchen away from an all out blood bath.

Most large businesses have a reluctance to change or only implement change when a major IT failure forces them to evaluate their current practices. Change is risky, costs money and nobody ever got fired for keeping things as they are. Corporate inertia is a terrible hurdle to overcome.

I've been in places where a Remedy Form for requesting a server outage is 7 pages long and takes 12 iterations before it's accepted. I've been in places where IT support is out-sourced and someone signed off on a 2 week SLA for changing a firewall configuration. Everyone hates those processes and for good reason.

As Patrick has said "collaboration is hard". I think the rise in automation (CI/CD, Chat Bots etc) can go some way to easing the burden of such collaboration. But automation can only go so far - if the process sucks, all the automation and collaboration in the world is not going to magically deliver improvement.

 

This is the most practical and realistic piece I have seen around DevOps. It is a huge lift that is so drastically overlooked that it can be mind boggling. Doing this right can change a company forever and similarly doing it wrong can result in so much churn it can sink a company.

 

TBF I work with SME's & I think devops (like healthy living) means different things to different people.

Once you have your infrastructure ready to deploy, backup, test & migrate, roll creds via code (most of which is borrowed from open source); you're done until you grow significantly in the SME space.

Enterprise DevSecOps is so far off these business radar, and capability compared to a bank or other industries if through nothing but price, that being able to delete your application, log events and statuses, whilst not losing or sharing pii is where its at.

I'd agree you're never done, but unless its a tech startup that's funded and regulated, iterating below the application will still be a once per {insert-period} activity resisted or classed as "we're done", or "I thought we'd tackled" no matter how large an emphasis is placed.

I know people working for HUGE payment processors that deal with these SMBs when they breach and get fined. Its the same every time "we didn't know"... " how to avoid?" :ostrich impression: "that's impractical"

I'd just like there to be more practical advice for the mom-and-pop shops, or innovation so they can get the ford focus, or kia roadmap for decent development with some things borrowed from cloud & automation

 

thanks for your post . good info devops article nice to read .

 

Very nice post on DevOps here and thanks for it .I always like and such a super contents of these post. Ifyou want to know more about DevOps and its tools by the following url : goo.gl/1T8W4q

 
 

Great article!! Check out datavisitor for Devops career path for non technical people.

 

Excellent article. After visiting many websites I got good information from your post.
For more information about DevOps course