DEV Community

Facundo Gauna
Facundo Gauna

Posted on • Originally published at gaunacode.com

Here are my "lessons learned" after 5+ years of DevOps consulting

#devops #career

Have you wondered what it's like to be a consultant before? Are you running into challenges in growing a DevOps culture at your job? I will share it all, and it will be a long blog post. To be honest, I don't know why I'm doing this; I just feel like it needs to be done.

I have been a consultant for boutique, medium-sized, and large consultancies. Before consulting, I was a developer. All of my consulting experience has been in helping companies adopt DevOps practices. Because I am a consultant, I get exposure to many more companies than I would if I were just an employee at them. So, I hope that if you don't know much about consulting, you get a view into my world, and if you're an existing consultant, then maybe I can help you with a challenge you've had.

Table of Contents

Before we get started

I was inspired at my day job because lots of our recruits don't have previous consulting experience, so I've been reflecting on what it means to be a "consultant." I thought that others could benefit from the lessons I have learned over the years. I am also doing this for myself because it's helping me reflect.

These opinions are my own, dear employer, don't fire me. 🙃

For context, I live and work remotely for clients in the United States. I know there's an international audience through this medium.

What this blog post is not about

In my view, some "consulting" companies are staff augmentation companies. In other words, the "client" is a company that needs people with certain skills. The "client" can't find people because they don't have a strong brand or recruiting capabilities, their payscales don't allow for competitive compensation, or the company likes to spool up or down the workforce with contractors, etc. When a consulting company does a "staff augmentation" project, the "client" does not expect strong advisory services.

I believe a "consultant" is an advisor and change agent looking to help the "client" achieve their goals. So, in this post, I will share my opinions about projects where I was strictly in that role - an advisor.

Lastly, I consider "DevOps" to include people, process, and technology aspects. So, this will not be a blog post around tips and tricks for creating pipelines or cranking out containers. Spoiler alert: Technology is the easy part. 😱

Consultant Glossary

  • Statement of Work (SOW) - The contract for a project between a consulting company and the client organization.
  • Project Plan - A plan on how to deliver the project. Sometimes very Ghantt chart esque.
  • Client Organization - The company that hires a consulting company.
  • Client - The main point of contact or stakeholder that works for the client organization.

1. Do the hardest thing first

Often, as engineers, we tend to want to put our heads down and crank out code or fix technical issues. But we're human beings, and as humans, we are highly social creatures. There is no "avoiding people." And people also create processes to get stuff done.

So, do the hard things first. You're probably dreading the hard things, and they will most likely be problems related to "people" or "processes."

So, for example, here's a list of things I watch out for.

  1. Go to production - Deploy to production as early as possible. Just because you "deploy" to production doesn't mean you have to "release" it to your users. More on this later.
  2. Have that security review with Central IT - If you're part of a big traditional enterprise, you are likely required to have a "security review" with enterprise architects and security architects. Do it as early as possible, be ready for it, and sometimes you'll be surprised to find that they're more straightforward than you made them out to be in your head. However, you want to avoid the scenario where you wait until the big production deployment towards the end of the project to have that security review. Then you receive feedback that will fundamentally change your architecture.
  3. Have that review with your enterprise architects - Like the previous point, propose a review at the beginning of the project in addition to the final one. The earlier, the better, so that you avoid surprises.
  4. Get the "process" stuff out of the way - Have to integrate with systems you find dumb, but they're crucial to your project? Do it soon. Don't run a project on assumptions that things will work. Validate those assumptions.
  5. Start that documentation early - Often, I have to provide long recommendation documents. So, I "write drunk" and "edit sober." Meaning writing the first draft without editing. If you try to write and edit, you will get stuck in writer's block or perfectionism.

Here's a strategy I borrowed from a client for greenfield apps - perform a walking skeleton production deployment early in the project. A walking skeleton is just a basic set of the infrastructure deployed to production with very little actual code. Also, a walking skeleton uses Infrastructure as Code and a CI/CD pipeline. The sheer nature of being in "production" will trigger all those processes you dread, and you can get them out of the way early. You will avoid nasty surprises and delays toward the end of the project. You will also gain a lot of practice and confidence deploying to production with the automation you established in the first place.

2. "Big bang deployment" is a bad word

Remember the "big bang" theory? There was a HUGE explosion, and then there was the universe. Deployments to production can be that way, too - a big bang deployment of infrastructure and code, chaos, and hopefully a working application.

You might be surprised, but so many companies today still wait until the end of the project to release to production. They hope and pray that a big bang will work. Sometimes, they ignorantly think nothing wrong will happen because of their skill and experience.

A big bang deployment is an explosion of Work in Progress (WIP). WIP is unfinished work. The work is unfinished because it hasn't been shipped to production, and you're not learning from it yet. The more WIP you have in the system at one time, the more risk.

Can you make a big bang less painful? Yes. I have had to suck it up and do big-bang deployments before. Here are some tactics that have helped me:

  • Infrastructure as Code at the beginning of the project - Less environment drift between non-prod, the better. It's going to alleviate surprises on the production side. But it won't stop them completely. If you start at the beginning of the project, you'll get a lot more repetitions with the process and have the training needed for the big day.
  • CI/CD pipeline at the beginning of the project - You become reliant on a pipeline to perform the deployments of the code. Therefore, you're less likely to have surprises on the big day.
  • Create a deployment plan and review it thoroughly - Create a simple document and list step-by-step all the things the team will do to perform the deployment. If you can, deploy a whitespace deployment in a non-production environment to validate the document. A whitespace deployment is just a deployment from scratch, so re-create the infrastructure and redeploy the code.
  • Break it up into chunks - Production "release" is not the same as a production "deployment." A release is when you make it available to your customers. So, if you deploy to production, you don't have to make it available to your customers yet. For example, you could deploy the cloud infrastructure, go to sleep, deploy the code, go to sleep, run some automated functional tests to validate, go to sleep, then perform database migration and do the DNS cutover.

Ideally, if I had a magic wand and I could get rid of Big Bang deployments, I would:

  1. Create the infrastructure via code at the beginning of the project.
  2. Set up CI/CD pipelines for my infrastructure and the application code at the beginning of the project. Then deploy to production frequently throughout the development of the app. Feature flags can also be very helpful here.
  3. Perform whitespace deployments occasionally throughout the project.

The last thing on this topic, you might think that big-bang deployments only affect companies that are not using Agile. In my experience, just because a company is "using" Agile or Scrum or SAFe doesn't automatically mean they are free from big-bang deployments. I've worked with zero companies that identify as using Waterfall, and I have run into many, many, many big-bang deployments.

3. Listen to the client and bring them into the conversation. Co-create with them. But there's a catch.

When designing a system, I like to bring the client into the design conversations. Maybe have "whiteboarding" sessions or just a series of meetings where we talk. At some point, I begin to form an opinion and have an idea of what I would like to recommend. So, I want to fail fast and validate my assumptions with the client. I also recognize that I might not have all the data points, so if possible, I create a simple draft document or PowerPoint stating options on a part of the system that could be designed. I present the options to a core client team with the goal of learning more. Often, they provide minor feedback or significant feedback. Sometimes the options get smashed together into a new option. Sometimes, we need to create an experiment to validate one of the options to address any concerns. The key is that the client is involved and therefore feels in control of the project's outcome.

In my observations, traditional consulting companies have different project delivery methods. They create a detailed project plan and weekly email reports and assign a project manager to the project. The plan has a clear "design/document/build" phase. Depending on the size of the project, there could be several months before anything is built!

What happens if there are gaps in the design document? Will the implementation go according to plan? Does the project need to halt because the design needs to be flushed out? I'm not sure, but it does sound risky, right?

If the project does get delayed, the client would not feel good about it. Even worse, if the design phase is dragged out until it's "right," it could mean months without actually building something or driving outcomes for the client. In my opinion, the client typically doesn't care about a recommendations document - it just doesn't drive the outcome for them at all. The document is a method for communicating design decisions and rationale.

Overall, I think one huge problem with the traditional consulting approach: the client was not involved in the design process.

How would you feel if you attended 10+ meetings talking about system design, and when you asked a question, the consultant said that the final answer would be in a recommendations document? Then, several weeks or months later, you receive over a hundred pages of the document. So, either:

  • A. You have no time to read it - You trust that there are appropriate recommendations based on the points you have made. Maybe you skim the document and can't find the things you're looking for, so you make a point to come back to it but never do.
  • B. You make the time to read it, and you're disappointed - The document doesn't have enough detail and rationale behind the decisions. Or maybe, there are contradictions between recommendations. Or, more frustratingly, the questions and points you have made during the discussions were never addressed. After you read more and more, you start to question, "this document is generic; it just states obvious things that we already knew."

Alright, so what's the catch to co-creating and designing with your client?

You can't hide.

If you're designing with them in the open, you will get to a point where you have to say, "I don't know." Or "in my experience, that hasn't worked well," or "I haven't tried this solution yet."

So ultimately, the client will know how much you know and how much you don't know. It will come out if you don't know enough about the subject you're talking about.

I don't recommend this approach to someone that doesn't have a lot of experience in the topic.

4. Know when to step back. Know when to step in.

Active listening is crucial. Stating opinions is important too. So, like any normal conversation, I think talking to a client should be an ebb and flow of listening and stating opinions. This "ebb and flow" is more
important the more people there are in the "room." I believe that the "ebb and flow" quality often translates to the customer experience. The reason is that sometimes clients bring in consultancies because they cannot work well with a different team; therefore, if you manage to have two or more teams talking and working together, that's a tremendous value.

Here are two extremes that I am not fond of:

  • Too much talking - Talking so much to "control" the conversation and "impress" the client. Often, it just comes across like someone likes to hear themselves talk. Besides, if you do all the talking, there's very little listening, and if you want to surely fail a project, that's to not listen at all.
  • Listening and punting - Too much of one thing is bad, and the same is valid for listening. Punting is when opinions are reserved until a later point and not "showing your cards." Being very introverted, I struggle with this.

5. The client doesn't give a shit about the SOW. The client wants results.

In my experience, it's normal for clients not to read the contract. Maybe, the stakeholder read it but didn't understand the technical fine points. Or maybe the stakeholder is not part of the day-day delivery of the project and did not share with the main people you're working with. Sometimes, I believe that they expect excellent customer service, and they trust that the consulting company understands their needs, so they never get around to reading it. Whatever the reason is, the SOW doesn't matter to them. What matters to them is that they can reach the outcome that they're looking for. It is very possible that you do everything in the SOW within the timeline, and the client is unsatisfied.

If the SOW does not state clearly the goals or outcomes of the engagement, ask the client. Sometimes the client will be a little annoyed since they would have had these conversations already during the pre-sales process. Still, it's better that they are annoyed that the project starts with complete misalignment from the beginning.

I'll share with you some high-level outcomes of my past projects so that you get a sense of how high-level we're talking.

  • "We acquired a company with an extensive software system, and their system is not up to our security standards. We want to redo everything using DevOps practices and Infrastructure as Code to replicate these practices for other systems in the future and cut the lead time to onboard new applications to the cloud."
  • "We want to modernize our monolithic application because we had a severe outage that lost us millions of dollars of revenue. If it happens again, it could be detrimental to the business. We feel this technical debt on the monolith is catching up to us; it's been growing for a decade."
  • "We want to find a new branching strategy because our teams struggle to deliver value consistently during their sprints. Every sprint, we also have crazy four-hour deployment meetings at 3 AM because usually, we have challenges deploying to production without an issue. We have other pains like code freezes against our QA environment, large deployment checklists, and long hours before the end of the sprint to try to get work done."
  • "We want to standardize our cloud infrastructure to cut down little decisions when we onboard applications to the cloud platform. By standardizing, we will be able to cut lead times to deploy new applications in the cloud and keep up with the demand of the enterprise so that we don't have to grow the cloud team."

Notice that I did not say, "we want ten Kubernetes clusters, twenty pipelines, and lots of shit doing fancy things."

6. Don’t hide behind the project manager, the salesperson, or the engineering director. Be vulnerable.

In my career, I have witnessed engineers that prefer to focus on the "technical" things while they lean on other teammates to do the "people" stuff. The same goes for consulting, and I have witnessed consultants that want to rely on the project manager to talk to the client. Heck, I have done this too. But, in my experience, this is what ends up happening:

  1. Project manager asks what the team has done and delivered.
  2. Engineers speak a bunch of technical gobbly doo.
  3. Project manager has a project update with the stakeholders and presents all the technical gobbly doo, but because they don't understand it, they don't explain it well.
  4. Client asks questions, and the project manager says, "I don't know," and asks the engineer during the meeting.

I believe it's much more effective for consultants to learn the soft skills necessary to say the hard things and communicate risks to the client. I'll give you an example of hard things I have said before:

  • "I am afraid that if we don't stand up this Kubernetes cluster soon, we won't have enough time to perform a load test with the application on it and, therefore, not be ready for the traffic spike in the high season."
  • "I am concerned that if we don't migrate this database to the cloud, then there's a risk that latency will be too large for the API. So, let's migrate it, but that will require us to complete that security review now."
  • "I know this project is really important to you. I hear what you're saying. Unfortunately, we won't be able to start XYZ until the week of A because we have to finish B and an upcoming vacation."

I won't lie to you - it takes time and practice. If you're early on in your career, start small. You'll be much more comfortable asking questions when you're a senior team member practicing this skill for years.

7. Working under SAFe is really really hard

Big picture diagram showing the complexity of SAFe 5

I'll be honest with you. I haven't taken the time to learn or understand the principles of SAFe. SAFe is short for Scaled Agile Framework, which was put together to help dictate how to do agile in a large enterprise. I can't speak to the details of exactly why it doesn't fit well with DevOps. But I can tell you my experience.

When I hear that a client is using SAFe, I bear for a long and slow burn-out-inducing project. It could be that 99% of my clients that have implemented SAFe have implemented it incorrectly, so it could be that it's not SAFe's fault. Based on opinions from others more well-versed in this topic, I don't think that's the case.

When I engage with a client organization that's typically already using SAFe, we have to use that delivery model to deliver a project. So, here are my observations about trying to deliver a project under SAFe:

  • Work to do work - So much work just to do work. Not only are there the typical Scrum ceremonies, but there's also lots of other planning to ensure that other's team's planning is planned effectively. In the end, it's a lot of planning work to prepare to do the work.
  • Super large meetings - To ensure that multiple teams plan effectively, they get shoved into a massive meeting. It could be a meeting with maybe ten or so teams or hundreds of teams. This large meeting is typically called "Program Increment" or "PI" for short. I have had clients with a PI planning event that's six hundred people large. Have you ever attended a six-hundred-person meeting? I have. Shoutout to Microsoft Teams for handling that massive scale.
  • It's just disguised Scrumfall - Scrumfall is when a team uses Scrum in Waterfall fashion. If you're "iterating" for nine months before you deploy to production, it's probably Waterfall, not Agile. With SAFe, there's so much emphasis on planning and the illusion of iterations that it's hard to deploy or deliver anything to production for the first time under this model.

Did you know that big tech companies don't use SAFe? Even just Scrum, for that matter?

So, do you have to do what big tech companies do? Nope.

But this data point probably means that you don't need SAFe to be able to be successful, right?

Before trying to undo SAFe at your company, read my other points.

So, it's likely that you'll have to learn how to make the best out of it. Here's what worked well for me:

  • Build excitement before a big planning increment - In SAFe, you have huge planning events once per quarter. If you can't convince another team to add some DevOps work during a program increment, then it's likely not going to happen. Therefore, before the big planning event, raise awareness before the big planning starts so that they'll add the work you're hoping to see onto their program increment. Do a brown bag, a demo, etc.
  • Re-adjust your pace to match the client's pace - There is only so fast you can move under this world. I have had to learn multiple times to let go and be ok with moving much slower than I would like.

Edit: Lastly, here's a great guide from Agile thought leaders.

Screenshot showing the title of the recommended guide about SAFe

8. Sunk cost fallacy is a strong force. Add ego, and you are screwed.

The sunk cost fallacy is when someone decides to keep going in a specific direction because there's been too much time, money, and effort put into the project. I can't tell you how often a client has chosen to continue doing something a certain way just because they've spent too much time on the implementation.

The ego is a self-protection mechanism that we all humans have. The ego jumps out to help our self-esteem when we feel threatened. Maybe, we feel incompetent if XYZ were true, or we feel like we lost control if XYZ were true, or our bosses won't like us if XYZ were true. When someone has an extraordinary amount of ego, especially when they choose to listen to information based on the status of the person giving the information, it's going to be hard. Add some sunk-cost fallacy, and it becomes a fast-moving training heading for a cliff.

Here are some examples:

  • "I went to an ivy-league school, and I created this fantastic library. We're going to keep using this library."
  • "Microsoft uses Azure Service Fabric, so we'll keep using Azure Service Fabric."
  • "Jimmy worked at Amazing Tech Company, so he's probably right on this. We're going to listen to Jimmy."

As a consultant, you can't control the client's decision. You have zero control. You only have the power of influence. If you run into someone with an extraordinary "ego," then the best thing you can try is to speak to his/her ego and provide information tied to status. In the end, if they don't value or respect your authority on the subject, they most likely won't listen. It might as well be an immovable object.

So, if I encounter a lot of ego and they're protecting something that cost them a lot of time and effort to implement, I pick a different problem to tackle. Usually, there are a lot of problems to solve, so this is not an issue. But if the problem they're protecting is crucial to the outcome for the client, then I'd have to lay it all out on the table and say the hard thing. In practice, I would probably escalate the issue and add other key personnel to the conversation to increase the stakes and the pressure.

9. Pick a hill to die on. Don’t fight all the battles.

If you picture a medieval times battlefield where armies fight close combat with swords, then the army on top of a hill usually has the advantage, and they try to defend the hill. The opposing army then fights uphill to defeat the enemy, but it will be more challenging. If you don't like the combative analogy, think of yourself as a love warrior trying to spread love 🙃. You will still only have enough energy to climb some of the hills.

When I start a project, I look for the "hills." The hills are challenging topics that will require a tremendous amount of emotional energy. Because of the energy, these initiatives will take, I pick one or two. Based on experience, I rule out some hills automatically because I know it probably won't help the project move forward.

Here are some of those "hills" that I always avoid:

  • Trying to convince the client to use something other than JIRA.
  • Trying to convince my client not to use SAFe for my project.

10. There needs to be trust between you and the client. If there’s no trust, they will not listen. In the beginning, they probably will not trust you. Get some quick wins to show value and you will earn trust.

When I make recommendations, I imagine having a "trust budget." Each big recommendation and big ask that will mean a lot of work for my client will consume from the "trust budget." Before I make these recommendations, I visualize how much of my trust budget I will consume. Often, I have a zero trust budget at the beginning of a project with a new client because I haven't delivered results yet.

So, if trust is earned via results, I quickly try to increase my trust budget so that I can make recommendations. Therefore, I am looking for some quick wins.

Quick wins are painful problems that are relatively easy to solve. When clients hear a low-effort solution to a nagging problem, they feel relief and excitement. And, if you're able to solve the silly problem along the way, then tell them afterward... they might make a statue of you outside their building.

Here are some examples of quick wins I have delivered:

  • Document the onboarding steps you're going through and deliver them as an onboarding guide or checklist for future engineers.
  • Add to their READMEs as you're learning about the systems.
  • Tweaking ugly deployment scripts to make them cleaner.
  • Provide easy recommendations on slashing cloud costs.
  • Provide a demo of a potential solution to a nagging problem, ideally within the project's first week.
  • Find a security vulnerability and raise awareness about it. Caveat: this only works if the company values security and doesn't see security as more work to do.

What if you can't find a quick win?

Then listen closely and ensure you'll solve the main problem extremely well. The key point is that when you deliver your recommendations, you want them to be heard. If you sense skepticism when talking to the client, that's probably an indication of a low "trust budget."

But skepticism is different than curiosity. Skepticism is "Do I trust this consultant that they can solve this problem? Have they done it before?". Curiosity is, "Okay, if this solution works, it would be amazing. How do I make this happen? What could go wrong?"

11. There’s no good and bad. Just effective or ineffective.

Don't be dogmatic. Don't hide behind "best practices." Explain why something is better or more effective.

I don't like "best practices." It implies that everyone should make this master list of things, and if you're not, then it's bad, and you're failing. They can also be a crutch to consultants.

I'm much more interested in the essence of a "best practice." Why is this a good idea? Why should your client do this? How much effort will they need to place to get the benefit?

For example, "it is usually best practice to usually keep secrets out of source control because if you decide to give read access to a repository to a contractor, then they can see those secrets and use them to gain access to an environment that they shouldn't have access to."

So, when you understand why something is considered "best practice," you can discuss risks vs. benefits with your client. If you're having trouble explaining why something is considered best practice, you probably don't understand it well enough yet.

12. When working with your team, call out ownership areas.

Some of the best projects I have delivered can be attributed to this single decision. Think of your team like a team of Avengers. Every person brings something different to the team, and each person has strengths and weaknesses. So, play to everyone's strengths.

For example:

  • Jimmy used to work as an ops guy, so he should probably own the monitoring recommendations.
  • Even though Anya is early in her career, she has a lot of drive. She can work on creating some demos that will take just as much time for someone senior to complete.
  • Carlos used to be a developer, so he should probably own the Infrastructure as Code (IaC) recommendations.
  • Sofia has the most experience, so she should probably design the Kubernetes cluster infrastructure since that's the most crucial part of this project.

When everyone agrees on what areas they will own upfront, it is much easier for the team to self-organize and react together. In contrast, if a team optimizes for "busyness" or time, then Jimmy, who doesn't have as much coding experience, might miss some crucial recommendations.

13. Sometimes, the customer experience matters more than the quality of the work. Sometimes, it’s more about how the client felt than how amazing the delivery was.

Although I cannot find the source, I heard of a study that found that average people thought the "best" doctors were the ones that had better bedside manners, not the more skilled ones. If this is true, it could also apply to many other aspects of life.

Receiving medical care can be a very vulnerable moment. Engaging with a consultant can also be a vulnerable moment since companies don't hire consultants when things are going well. Sometimes, the project that a client is sponsoring could mean a promotion or maybe the last ditch effort before they leave the company.

So, what is the client experiencing?

Do they feel listened to?

Do they feel understood?

Do they feel like they can trust their judgment?

I learned this lesson through a unique project. My consulting company has three teams working for a client organization. My team was the highest performer, and the client liked it best, but it wasn't because of me. It was because of my teammate.

At the time, I believed that delivering amazing client results was doing lots of great technical work. I put my head down, and I worked furiously to drive results. On the other hand, my teammate was not as ambitious; however, he did stop and strike up conversations with my client. He found out they both had military experience and loved Asian cuisine. Later, we ended up going to dinner with our client at a hole-in-the-wall yet amazing Korean BBQ place, and it was amazing.

I remember thinking, "Wow, what am I? Chopped liver? My client loves my team because of my teammate. I don't think it has anything to do with me."

My teammate just genuinely cared about the person in front of him. My client felt listened to and probably felt like they could talk openly when we had to make tough decisions at work. With time, after more dinners and lunches, we got to hear more stories and context from our clients. The relationship was growing into a friendship, and he likely felt very understood.

Wouldn't it be better to go through a vulnerable moment with someone you can trust?

Building a relationship with a client is very important.

14. Practice how you ask questions. The tone is so important. Avoid sounding combative. You’re trying to reduce the chances of people getting defensive.

You can't control how people feel. You can control how you say things in a way that minimizes the chances of someone being triggered and raising their defenses.

For example,

Consultant: Why are you doing it that way? That's not best-practice.

vs.

Consultant: Hmm. I am not tracking. Help me understand. What's the value of doing it this way one more time? I fear that we might not reach the project's goal with this approach.

When I think about how to ask questions, I think of two things:

  • I want my question to come from a place of curiosity.
  • I want to avoid the word "you."

To ask from a place of curiosity, I often have to take a deep breath, put myself in the client's shoes, be empathic for them, and assume that they are trying their best given their circumstances.

Also, by avoiding using the word "you" forces me to talk about my experience. No one can challenge how I feel or what I have experienced. It is my truth.

Don't get me wrong. I am still me, and I am still a person with flaws. I always have moments where I think to myself, "what the f***?." The difference is that when those feelings arise, I acknowledge them and take a moment to think about the person in front of me.

Side note: I was fortunate to work with an organizational psychologist at a previous consultancy. She trained me and fellow consultants to avoid certain words so we could speak more collaboratively.

Picture of a big poster that we used to have in the office
We had this posted in our office so that we would remember to use the vocabulary.

15. When people lose their cool and start yelling, you can be the voice of reason.

As a consultant, you probably don't get hired into totally functional companies. From time to time, you might be caught in the middle of an internal political battle where someone ultimately loses their temper. That's a moment where you can add a lot of value.

Here's a recount of something that happened to me:

Person 1: We can't design the network this way. It leaves XYZ vulnerable. We'd need another ABC service and another firewall.

Person 2: Ah ok. But this part of the design is ok, right?

Person 1: No, it's not. We can't link these networks together because XYZ is vulnerable.

Person 2 gets frustrated because he wants to go live with the project.

Person 2: We need to start doing things in a new way! We can't be doing data center crap in the cloud!

Yelling continues, then stops for a second.

Me: Yeah, Person 1, I understand what you're saying. What if we get rid of the issue? Because this is a greenfield deployment, could we say that workloads that don't meet these criteria won't be able to be hosted here? That way, we can ship and don't have to design for all possible use cases, especially legacy ones.

Honestly, I can't tell you if this conversation moved them closer to the goal. I can tell you from more than one account that sometimes groups bring you in because they feel pain points in "dealing with another team." They want someone to help them work with another team. So, if I didn't try to be a mediator, I would have had zero chance of moving the outcome forward. If they weren't working well together before they hired me, then they will probably not magically begin to collaborate, right?

The best thing you can do is try; if it doesn't work, it's not any worse than before.

16. You don’t need the client to like you. They don’t need to be your friend. Say the hard things.

I have to tell myself this all the time. I want people to like me, and I enjoy working with people that are having a good time. But, sometimes, what the client needs is not a friend. Sometimes, they need to hear what is wrong or what could go wrong - directly and clearly.

"If we don't do this soon, I fear we'll not have time to go to production."

"I know the burndown chart says we're tracking to finish the project; I strongly feel that we will need more time than our board says."

"I know you have spent a lot of time on your solution. I want to present you with the alternatives so you know our stance and the surprises that might come up. I would hate for you to continue your current path without knowing the risks. Ultimately, it's your decision, and if you choose to continue on this path, you know the risks."

So don't avoid saying the hard thing because you are afraid they won't like you anymore. Also, remember, watch your tone!

17. As a consultant, you have no control. It’s all influence.

Early in my career, learning that I had no control was probably one of the biggest lessons that took me the longest to learn. I kept getting frustrated working with some clients because they were not listening fully to my advice. I had to learn that I have no control over what the client decides or does.

I have no "skin in the game." I don't work for the company that they work for. I probably won't get fired if this project does not go well. So, it's only fair that if they hold the risk, then they decide, right?

Instead, the only tool you have is influence.

How do you influence?

Think of social media influencers like YouTubers. What do they do?

  • They share information in easily digestible ways.
  • They tell you about other products with honest reviews.
  • They share their experience and what's worked for them.
  • They connect you with other great ideas and people.

All those things apply, minus the YouTube channel.

18. Do not word vomit. Use "top-down" communication.

Have you ever sat through a painful presentation where an engineer is presenting to upper management, and management seems disinterested? Was it painful? Did it seem like it was dragging on?

McKinsey is a world-renowned management consulting company. McKinsey uses "top-down" communication. Instead of telling a long story and finally providing the key point at the end, present the key point first and then explain.

This blog post is an example of that kind of communication. The key points are delivered as headings. The points might not make sense unless you read the explanations like this one. But, you have the option to determine if the key point is something you're interested in, and I'm not holding your attention hostage.

I think this blog post in story form would be much more difficult to read. You would have to read everything to get the twenty-eight lessons I am sharing.

As an engineer, you might cringe, but I find PowerPoints very useful. I find them useful because they're a tool that allow me to help break down complex ideas into small chunks and communicate them effectively.
For example:

  • When creating PowerPoints, I create a slide to share the main recommendations in the beginning of the presentation.
  • I try to keep six items on the slide at once. Items include images and words. When I do this, it typically leads to people paying more attention to what I am saying because they are not caught reading slides. It also forces me to explain the key point and focus on it.

If you want to learn more about the McKinsey method of communication, here's a great video from an ex-McKinsey consultant.

19. Who are you talking to? IT doesn't want more work, and developers want to ship more.

Unless you have decades of industry experience and managed not to become a manager, you probably don't have experience being a professional system administrator and developer. So, you likely only have experience in the Dev or the Ops in DevOps. If you're a junior DevOps engineer, you have a serious disadvantage.

Part of playing politics and reaching agreements is figuring out what different people want to achieve.
Very generally speaking:

  • Developers want to ship more value. Developers at large enterprises they're often frustrated with all the processes that prevent them from shipping and how little control they have over infrastructure, even in the cloud. I was a developer at a large enterprise, and I felt this pain point deeply.
  • IT wants less work to do. They include system administrators, identity admins, network engineers, infrastructure engineers, cloud admins, etc. They're usually underwater with work and often on call when things go wrong. They're responsible for large infrastructure and don't want to be called at night. Making things standardized makes their job easier. Also, if they're extremely busy, they will sometimes opt for tried and true technology and avoid new trends. Lastly, they also really care about top-down governance because they don't want people they don't know to cause more work for them.
  • Security - They feel pressure to keep the company safe from breaches. There are generally different specialties within security. For example, some security engineers focus on network security, some on monitoring and alerting, and some on application security. They know that they are often seen as the folks that halt projects.

If you have a different perspective on each group of people care about, that's ok. However, let's leave out security for the following metaphor; otherwise, it is too complicated.

Imagine that your role as a DevOps enabler is to be a bridge for "work." At one end of the bridge, there's "development," and at the other end, "operations." You have made your customers happier when you ship one unit of "work" across the bridge.

So, to become an effective bridge, you must know a bit about "development" and "operations." If you started your career as a developer, then you probably have gaps in operations, as I did. If you started your IT career, you probably have development gaps. If you started your career as a junior DevOps engineer, you probably have gaps in both realms. The key here is to know enough about each role so that you can speak their language. Being able to speak about what someone cares about is a great way to build trust. The more you can speak in someone's language, the more they will think, "you are one of us." And in dysfunctional environments that need the most help, there's usually an "us vs. them" mentality. No pressure. The better bridge you can be, the better advisor you will be.

So truly, here are some of my standard tactics to build more trust:

  • Go out of your way to add guardrails on behalf of IT. Guardrails are often policies that prevent people from doing insecure or unapproved things. Examples of guardrails are Azure Policies, HashiCorp Sentinel, and AWS Policies. After adding guardrails, you can then turn around and try to give developers more autonomy because there are guardrails that disallow them to do bad things.
  • Go out of your way to increase security before the security engineers say something. By increasing security preemptively, you can gain favor and help design a solution that is not a hindrance.
  • Show developers Kubernetes, Infrastructure as Code, and Application Performance Monitoring (APM) tools. They will be often be interested in doing little bit of "ops" work so that they can ship more.

20. You are a doctor

How would you interact with people if you thought of yourself as a doctor?

Would you try to find the root cause of a symptom instead of pushing pills first?

In this case, pills are tools. If you push tools all the time, you're not going to solve the root problems. Sometimes organizations require a lifestyle change, which means a process or culture change.

So, think about the care plan for the organization. It could start with a tool to alleviate some profound pain but then think about process improvements to solve the deeper issues.

21. You have a hidden force acting against your relationship with the client. It's hourly billing.

Consulting companies are businesses. Most consulting companies charge their clients by the hour. There is usually a margin between what they pay the consultant and what they charge the client. It's that simple.

So, because of the billing method, a consulting company makes more money when:

  • Projects take longer, even if they are delayed.
  • Solutions are more complex because they will take more time and/or people.
  • Client organizations cannot maintain a solution, so they must continue hiring the consulting organization.

I'll be honest. I find these to be extremes. I wouldn't work for a consulting organization that's doing these practices on purpose to make more money.

And, humans can be exceptionally good at smelling intent.

Also, we humans can be very sensitive to perceived ill intent. Think used car salesman.

For example, one time, I was working with an excellent client. We had a great relationship. They had asked for a follow-up engagement and given me a list of items to help them with. However, it was in the middle of a heavy sales process change, and I needed approval from general managers and go through a "deal review." I was so focused on getting approval and fighting the spreadsheets that I lost sight of the hourly rate we were charging the client before the process changed. I was trying to get approval so we could start soon, but the new rate was considerably higher. We sent the proposal to the client; they expressed that they had sticker shock (the price was extremely high). The relationship ended.

I believe that they thought that because we did a good job and they were asking for more help, we were being bold and raising the rates because they needed us. In reality, I was scoping what they had asked for and made a mistake carrying the proposal through the heavy-handed sales process.

The lesson to be learned is to remember what your company is incentivized to do and consider how your client will feel. Don't overcomplicate solutions, be transparent about delays, communicate early and often, and be mindful that a client must maintain your solution, show urgency, etc.

I'll leave you with this: imagine you hired a good plumber to fix a leaky pipe and he was charging you $150/hour. How would you feel is he was taking his sweet time and moving slow and looking ineffective? If you had a low budget, I think that would give anyone a little anxiety.

22. Virtual desktops over laptops

If you're a consultant, you might like this one. Request a virtual desktop instead of a laptop when starting a project that's less than a year. A virtual desktop allows you to easily switch between your company's and the client's communications without needing a fancy KVM.

Side note: here's a product idea: a very fancy KVM for consultants. Before giving up and accepting virtual desktops, I vigorously searched the market for the perfect KVM that would be supported by MacOS and Windows so that I could keep my peripherals plugged in one place and switch between computers effortlessly.

Consultants often have to get access to the client's systems to perform the work. Many companies have security postures that don't accept using the consultant's workstation, hence, bringing their own device. Instead, many enterprises like to provide laptops to contractors so that they can connect to the network from a device they can trust and control.

Well, if you're a consultant, these devices can pile up. I have an ugly laptop that's been collecting dust next to me for six months after I finished the project. 😊

I was once doing a project with a client that also hired AWS Professional Services (AWS consulting). I overheard the AWS project manager tell the client that using virtual desktops instead of laptops was part of their contract. I found this interesting because it took me two months to get set up with a laptop. It took the AWS consultants two weeks to get set up. Then, I started asking my clients for virtual desktops for short/medium projects.

By using VDIs, I was relieved that I didn't have to agonize anymore about having all this clutter and physical devices everywhere. Also, starting the project with virtual desktops is often much faster than with laptops.

I hope this doesn't happen to you, but I once had to make a round-trip of seven hours to go to a location to get a proprietary badge and then drive another nine hours round trip to pick up a laptop. The badge unlocked the laptop. Then, another time, I had to drive an hour each way, sit outside a client's branch location at a Chinese fast food place, get on the client's Wi-Fi, then talk to the IT help desk for two hours to get my awesome laptop set-up.

23. Juggling multiple emails is a necessary evil

Another tip for a new consultant.

On top of getting a new laptop from clients, we are often given an email address by the client. So on top of our actual work email addresses, we end up with all these other identities and inboxes for each client.

At the beginning of my consulting career, I tried hard to solve this problem. The reality is that many enterprises lock down any customizations to their emails so that you can't use external OAuth applications. Heck, I had one client that monitored their emails so closely that I was strongly warned that if I ever forwarded an internal email to an external account, an automated system would pick it up and fire me. I was also strongly advised not to print anything because I would be automatically fired.

You will have multiple emails and identities. Just accept it. It's the consulting tax. 😀

24. Windows is sometimes a necessary evil

Last tip for an existing consultant, especially one that hates Windows. 😄

To be clear, I like Windows, even though I'm in the DevOps field. I'm writing this blog post from a Windows 11 laptop. 🤗

However, I wouldn't say I like Windows as a typical enterprise setting, and that's because of all the controls and bloatware that enterprises install. I bet everyone hates slow laptops and funky issues due to bloatware.

Just like you can ask for a virtual desktop, you can also ask for what operating systems they support. Rarely do enterprises support Linux for workstations. Sometimes, you might get lucky, and an enterprise supports MacOs. Most of the time, enterprises only support Windows.

25. KISS is relative. Simplicity is clear; it's not lazy.

Many engineers like to say Keep It Simply Stupid (KISS). I agree. But I think simplicity is relative.

What you might find simple, I might not find simple. What I could find simple, the client might find complicated. What the client might find simple, I might find an abomination.

I think simple takes work. I don't think arriving at the first solution is often the "simple" one. Doing the least amount of work can be "lazy." So, I'm going to do this... I'm going to give you a Steve Jobs quote. 😀

“Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains.” - Steve Jobs

There have been times that I worked hard to make a solution "simple" for clients. Here are some of the attributes I'm looking for:

  • Simple is clear - When the client reviews the solution, does it have an "aha!" moment? Will it click? Many consulting solutions go to die because when the client looks at the solution, they find it too complex and don't have the time to reverse engineer it.
  • Simple is maintainable - Assuming that the client will work to learn the foundations of technology, will they know enough to maintain and extend it? One time, I had a client demanding that we undo a CI/CD pipeline written in YAML because they wanted to use the old and comfortable GUI-based pipelines in Azure DevOps. I think this was lazy on the client's part because they wanted to do what they've always done instead of exploring an industry practice. But, if our client had learned the essentials of building a YAML-based pipeline, could they grab our pipeline and understand it enough to extend it?
  • Simple is not a Rube-Goldberg machine - Is your solution starting to feel like a Rube-Goldberg machine? Maybe, it's simple on its face but complicated behind the scenes. For instance, I was reviewing an application that my client had built, three simple pages. It was a user sign-up, a login, and the main page. Behind the scenes, there were like twenty microservices! It took a lot of cognitive load to understand what was happening in the backend. Oof.

26. Avoid designing the bike shed

I learned this one from a co-worker recently. I think it applies to a lot of organizations.

What is described in this Wikipedia article, and what I have experienced, is that often we can easily focus on trivial things. To "avoid designing the bike shed" means that if you're designing something important like a nuclear power plant, then don't place as much importance on designing the bike shed next to the power plant.

Recently, I was on a project to design a very important enterprise-scale landing zone for the cloud, meaning the cloud foundation. The client was interested in discussing naming conventions and spending time on them. Naming is hard. I get it. So, I'd rather use a naming convention that a reputable source created rather than waste precious mental energy creating my naming convention that I'll probably get wrong.

Be aware of this for yourself, your teammates, and your clients. You might be stuck discussing something that will not improve the nuclear power plant.

27. Confidence is different from assertiveness. Confidence comes from practice.

Have you met someone that seems like they're exuding confidence? I'm not that person, and I've always envied that to some extent.

Instead, I'm naturally quite the opposite. I have a lot of self-doubts, and I double-guess myself. So, I've asked co-workers, "you sound so confident; how do you do it?."

Sometimes, some people may sound confident, but inside, they are still scared. Their physiological responses are ones that you might not see. For example, when I get nervous, you can tell. My ears turn red, I start to sweat a lot, and I start to stutter and think slower. So, some people might be nervous, yet they don't show those clear signs, and then you might make up a story in your mind that they're confident because they sound so assertive.

The true solution that I found that works for me is practice. The more I do something, the more confident I get. So, I accept it and give myself permission to be nervous when I'm not great at something yet. If I keep showing up and doing, I will eventually get better and be much more confident.

Here's a book that helped me and might help you: The Confidence Gap: A Guide to Overcoming Fear and Self-Doubt.

28. How you treat others is a reflection of how you treat yourself

Do you remember a time that you worked with someone that was really critical?

Maybe they weren't criticizing you, but you were listening to how they criticizing someone else. I've come to know someone with this tendency, and when I hear what they think of themselves, it's rough. I wouldn't want to be in their brain. There's a lot of stress and hate, so if they're hard on others, they're likely even harsher on themselves.

Watch how you show up with others. If you're passing judgment, you're likely to pass a lot of judgment on yourself. Be kind to yourself.

Also, it works the other way around. When working with someone overly critical of you, try thinking about how much harder they probably are on themselves. If I hear harsh criticism from someone, I find it useful to think about their inner struggle. It helps me center myself and lower my defenses because I realize they're also struggling.

Thank you for your attention!

I know there's so much content out there. I'm honored if you've read some or all of this article. Remember, I'm just a random dude from the internet, and some of my lessons might or might not help you.

Top comments (0)

Read next

bukton123 profile image

มาทำความรู้จัก Nautik เครื่องมือสำหรับจัดการ kubernetes ผ่าน iPhone, iPad เเละ Mac

BukTon -

dm8ry profile image

Finding Tables Without Primary Keys in PostgreSQL Databases

Dmitry Romanoff -

kungfoosian profile image

How I built the infrastructure for my cloud resume

Anthony Vo -

willvelida profile image

Integrating Azure Policy in your AKS cluster using Terraform

Will Velida -