DEV Community

Cover image for Dirty secrets of DevOps
Chris Dodds
Chris Dodds

Posted on

Dirty secrets of DevOps

I've read dozens of DevOps success stories, tales of bold IT leaders transforming their business and steering big corporate ships into the future. It's hard to avoid all these stories about "DevOps transformation" and how some guy named Jim pulled the IT department out of the stone age. The trades, blogs, and conference presentations are filled with them.

No one talks about the failures though, very few even write about their struggles implementing DevOps. It's all sunshine and rainbows, which sucks, because that isn't real.

Success teaches us little other than survivorship bias and how to feel bad about what we haven't achieved. Failure and hardship are where we learn. That's where the good, meaty stuff is.

So here are a few dirty secrets of DevOps.

Most companies that say they are "doing DevOps", aren't.

Because of all the success stories (real or imagined) that have wiggled their way into the minds of CXOs, "we should be doing DevOps" became an empty corporate directive that inspired thousands of executives to start calling their IT infrastructure groups "DevOps" instead of "Infrastructure" or "Systems" or "Operations".

Unfortunately, this seems to often play out as "We've renamed the group, so we're going to be letting most of the team go, because we're DevOpsing all the things now and being lean and mean. Also, the developers are still a separate group and they'll be throwing more stuff over the wall to you."

So you end up with a few overworked, traditional Ops folks trying to keep the wheels on the bus with zero changes to the way work is managed or how the IT group functions day-to-day. Their manager is shouting down the hall about automation while the poor Ops team is trying to pivot a SAN-installing, server-racking skill set into something that looks like a cave-man version of coding.

The only metric that improves in this scenario is "personnel cost", and only temporarily because burnout and churn spike, driving up staffing costs a few quarters down the road. But it looks good long enough for someone to say "See, we did it!" and feel validated.

Even if you get the "IT folks" on board, getting plugged into the business so DevOps practices can benefit other groups and the overall bottom line comes with its own challenges.

Fixing this issue requires a lot of skill managing upward and sideways. Often times, it's not worth trying to change and moving on is a better option. Your mileage may vary.

Implementing DevOps is Really Fucking Hard™

DevOps is all about people and process, getting everyone working together to do fewer dumb things, and smart things faster.

Historically, getting people to work together and not be jerks to one-another has been a bit of a challenge. Humans achieve awesome things when we collaborate (like spaceships and lasers), but we usually suck at working together. Because of that, I'm always impressed when I come across an excellent people-whisperer, someone who can motivate different groups to work towards a common goal without burning down each other's village.

Problem is, there's like five of those people on the face of the planet and chances are, they don't work for your company. You might have lucked out and have 1 or 2 folks who are kind of OK at people-wrangling and peace-keeping, but most businesses (especially the bigger ones) seem about three seconds and a passive-aggressive sticky note in the office kitchen away from an all out blood bath.

Assuming you can get people working together, you're now faced with the challenge of implementing process. You probably have one person on your team who loves process. Everyone else hates process and that person.

You're never finished

There's no such thing as "we achieved DevOps". It's a practice like healthy living or Buddhism. There has to be a champion(s) on your team who pushes every day to make things better.

When someone talks about DevOps success, what they're really talking about is achieving flow, that there is a functional work process in place that is continually measured and improved upon. It's an ouroboros value pipeline.

That's not something you can arrive at and stop tending to. Without constant care and feeding, the processes you worked so hard to implement will start to die off.

No champion, no DevOps.

All that being said...

None of this means that DevOps isn't worth doing, just that you need to be realistic about the challenges you're going to face. I've leaned on hyperbole pretty hard to swing the pendulum away from sunshine and rainbows, because reality is somewhere in the middle.

Getting Dev and Ops (and Security), groups who have traditionally walked down the hall waving their middle fingers at each other, to 1.) work together and 2.) implement and adhere to process, will likely be one of the most difficult things you've attempted in your career. You have to put a lot of work into making the right things the easy things, reducing friction wherever you can. Setting mandates or badgering doesn't work, you have to sell the value.

Getting top-down buy-in (and understanding!) of true DevOps/Agile practices is hard. It requires reorganizing groups and a sustained sales pitch to all involved. The need for this trails off a little once the business and IT staff start seeing value, but expect it to be a long, sustained effort. I'm always a bit dubious when I hear something like "we transformed IT in three months" - either that group really has their shit together, someone is lying, or we're not using the same dictionary.

For practitioners and evangelists, these are the things we need to start talking about more. There's a slick consultant vibe that's weaved throughout discussions about DevOps that glosses over the practical and prescriptive. Too many of these conversations focus on high-level what-to-dos and not enough on concrete how-tos and context, especially when it comes to people-centric issues.

Originally posted on chrisdodds.net

Oldest comments (23)

Collapse
 
ben profile image
Ben Halpern

We're not big enough to bother distinguishing DevOpsy work too much from non-DevOpsy stuff, so we don't really think in some of these terms, but this stands out:

You're never finished

There is definitely an urge at every step in the process to feel like "Okay, once we get these few things in place, everything will be way smoother and automated and we can just focus on building features" but that day isn't going to come. It's not even going to come in the sense of "okay this portion of the team maintains the processes and the other folks are free to proceed uninhibited."

It's going to be the case that we are always going to be refining and evolving these things, and hopefully preemptively because we don't want to get caught off guard by a big shift.

Collapse
 
liquid_chickens profile image
Chris Dodds

I've run across a few really smart managers who build refactoring time into all their sprints that covers both core code and automation exactly for those reasons.

Collapse
 
aptituz profile image
Patrick Schönfeld

Thanks for your post.

I was really curious what would come, when I started reading your post, since you started your post like you wanted to say that and how DevOps sucks, while you actually said it's hard to implement and it's not an end in itself.

I'm not convinced that the whole Internet is "all sunshine and rainbows" when it comes to DevOps, but I can totally relate to the point you actually made.

As a matter of fact DevOps is about collaboration and collaboration is hard. Full stop.

Best Regards,
Patrick

Collapse
 
philgibbs profile image
Phil Gibbs

Excellent post Chris - some genuine LOL lines, especially this one:

...most businesses (especially the bigger ones) seem about three seconds and a passive-aggressive sticky note in the office kitchen away from an all out blood bath.

Most large businesses have a reluctance to change or only implement change when a major IT failure forces them to evaluate their current practices. Change is risky, costs money and nobody ever got fired for keeping things as they are. Corporate inertia is a terrible hurdle to overcome.

I've been in places where a Remedy Form for requesting a server outage is 7 pages long and takes 12 iterations before it's accepted. I've been in places where IT support is out-sourced and someone signed off on a 2 week SLA for changing a firewall configuration. Everyone hates those processes and for good reason.

As Patrick has said "collaboration is hard". I think the rise in automation (CI/CD, Chat Bots etc) can go some way to easing the burden of such collaboration. But automation can only go so far - if the process sucks, all the automation and collaboration in the world is not going to magically deliver improvement.

Collapse
 
vedagayathrihyd profile image
Veda Gayathri-India's Largest Pooja Booking Portal

Title: Mastering MuleSoft Security: Best Practices for Safeguarding Your Integration Platform

In the ever-evolving landscape of digital transformation, MuleSoft has emerged as a leading integration platform, empowering businesses to connect applications, data, and devices seamlessly. However, in this interconnected ecosystem, security becomes paramount. With cyber threats on the rise, safeguarding your MuleSoft environment is essential to protect sensitive data and ensure uninterrupted operations. In this comprehensive guide, we delve into the intricacies of MuleSoft security, outlining best practices and strategies to fortify your integration infrastructure.
SAP Mulesoft institutes in Hyderabad

Collapse
 
lewiscowles1986 profile image
Lewis Cowles • Edited

TBF I work with SME's & I think devops (like healthy living) means different things to different people.

Once you have your infrastructure ready to deploy, backup, test & migrate, roll creds via code (most of which is borrowed from open source); you're done until you grow significantly in the SME space.

Enterprise DevSecOps is so far off these business radar, and capability compared to a bank or other industries if through nothing but price, that being able to delete your application, log events and statuses, whilst not losing or sharing pii is where its at.

I'd agree you're never done, but unless its a tech startup that's funded and regulated, iterating below the application will still be a once per {insert-period} activity resisted or classed as "we're done", or "I thought we'd tackled" no matter how large an emphasis is placed.

I know people working for HUGE payment processors that deal with these SMBs when they breach and get fined. Its the same every time "we didn't know"... " how to avoid?" :ostrich impression: "that's impractical"

I'd just like there to be more practical advice for the mom-and-pop shops, or innovation so they can get the ford focus, or kia roadmap for decent development with some things borrowed from cloud & automation

Collapse
 
connor11528 profile image
Connor Leech

Isn't devops mostly about writing Chef/Puppet scripts and setting up AWS infrastructure? It'd be helpful to know about the field for those of us not in it

Collapse
 
yakhyadabo profile image
Yakhya DABO

DevOps is about :

  1. Culture (Collaboration, Communication, Organisation, ...)
  2. Processes (Continuous Delivery)
  3. Toolset (Configurations Management, Cloud computing, Container technologies, ...). And here's where you have Ansible, Terraform, AWS, Docker, Vagrant, ...
Collapse
 
kylegalbraith profile image
Kyle Galbraith

This is the most practical and realistic piece I have seen around DevOps. It is a huge lift that is so drastically overlooked that it can be mind boggling. Doing this right can change a company forever and similarly doing it wrong can result in so much churn it can sink a company.

Collapse
 
bgadrian profile image
Adrian B.G.

UU dirty secrets eating popcorn

Collapse
 
hopesuresh1 profile image
hopesuresh1

Very nice post on DevOps here and thanks for it .I always like and such a super contents of these post. Ifyou want to know more about DevOps and its tools by the following url : goo.gl/1T8W4q

Collapse
 
dreamzdaisy2 profile image
daisyridley

Great article!! Check out datavisitor for Devops career path for non technical people.

Collapse
 
sudamjena14 profile image
SUDAM JENA

Excellent article. After visiting many websites I got good information from your post.
For more information about DevOps course

Collapse
 
trainingsexpert profile image
onlinetrainingexpert

thanks for your post . good info devops article nice to read .

Collapse
 
vedagayathrihyd profile image
Veda Gayathri-India's Largest Pooja Booking Portal

Auditing and Monitoring:
Snowflake provides robust auditing capabilities, allowing organizations to track user activities, data modifications, and system events in real-time. By maintaining comprehensive audit trails, organizations can identify security incidents promptly and adhere to compliance requirements.

Data Masking and Redaction:
To prevent sensitive data exposure, Snowflake offers data masking and redaction features, enabling organizations to conceal confidential information dynamically. By applying masking policies based on predefined rules, Snowflake ensures data privacy and confidentiality.

Best Practices for Snowflake Security:

Implement Strong Authentication Mechanisms:
Enforce multi-factor authentication (MFA) and single sign-on (SSO) to authenticate user identities securely. Utilize strong passwords and regularly rotate access credentials to mitigate the risk of unauthorized access.

Encrypt Data Across the Entire Lifecycle:
Leverage Snowflake's built-in encryption capabilities to encrypt data both at rest and in transit. Implement key management best practices to safeguard encryption keys and ensure data confidentiality.

Enforce Least Privilege Access:
Adopt role-based access control (RBAC) to enforce least privilege access principles. Define granular access policies based on user roles, limiting access to data and functionalities essential for their responsibilities.

Monitor and Audit User Activities:
Enable comprehensive auditing and monitoring to track user activities, data modifications, and system events effectively. Leverage Snowflake's auditing features to generate audit logs and alerts for suspicious behavior or security incidents.

Regularly Review and Update Security Policies:
Stay vigilant against evolving threats by regularly reviewing and updating your security policies and configurations. Conduct periodic security assessments and audits to identify vulnerabilities and implement remediation measures promptly.

Educate Users on Security Best Practices:
Promote a culture of security awareness among users by providing training on security best practices, data handling guidelines, and regulatory compliance requirements. Encourage users to report any security incidents or suspicious activities promptly.

Conclusion:
Securing sensitive data in the cloud is a continuous process that requires a proactive approach and adherence to best practices. By leveraging Snowflake's robust security features and implementing the strategies outlined in this guide, organizations can fortify their data defenses and safeguard against potential threats effectively. By prioritizing security, organizations can maintain trust with customers, comply with regulatory standards, and mitigate the risks associated with data breaches. Master Snowflake security today and protect your most valuable data assets with confidence.
Snowflake institutes in Hyderabad

Collapse
 
vedagayathrihyd profile image
Veda Gayathri-India's Largest Pooja Booking Portal

Best Practices for MuleSoft Security

  1. Implement Multi-Factor Authentication (MFA)
    Enforce multi-factor authentication for accessing MuleSoft resources, adding an extra layer of security beyond passwords. By combining factors such as passwords, biometrics, and one-time passcodes, MFA mitigates the risk of unauthorized access and strengthens authentication mechanisms.

  2. Secure API Endpoints
    Apply security policies to API endpoints to control access, enforce rate limits, and validate incoming requests. Implement OAuth tokens for API authentication and leverage API gateways to centralize security policies and manage access control across APIs.

  3. Regular Security Audits and Penetration Testing
    Conduct periodic security audits and penetration tests to assess the effectiveness of MuleSoft security controls and identify potential vulnerabilities. Addressing security gaps promptly and implementing remediation measures enhances the resilience of your integration infrastructure against evolving threats.

  4. Stay Updated with Security Patches and Updates
    Keep MuleSoft runtime environments and components up to date with the latest security patches and updates. Timely patch management ensures that known vulnerabilities are addressed promptly, reducing the risk of exploitation by malicious actors.
    SAP Mulesoft online taining in Hyderabad

Collapse
 
vedagayathrihyd profile image
Veda Gayathri-India's Largest Pooja Booking Portal

Title: Fortifying Your Data Fortress: A Comprehensive Guide to MuleSoft Security Best Practices

Introduction:
In today's interconnected digital landscape, where data flows seamlessly across applications and systems, ensuring the security of integration platforms is critical. MuleSoft, a leading integration platform, facilitates the seamless exchange of data between disparate systems, enabling organizations to unlock new efficiencies and drive innovation. However, as data becomes increasingly valuable and regulatory requirements more stringent, safeguarding the integrity and confidentiality of information flowing through MuleSoft instances is paramount. In this comprehensive guide, we explore MuleSoft security best practices, empowering organizations to build a robust defense against potential threats and vulnerabilities.
SAP Mulesoft Training in Hyderabad

Some comments may only be visible to logged-in visitors. Sign in to view all comments.