DEV Community: Jesse DeRose

Beyond Cost Optimization: How to Solve Your FinOps Challenges

Jesse DeRose — Tue, 16 May 2023 18:37:06 +0000

Introduction

As engineering leaders, we glean insights from other industries to improve the way we work. It's all about thinking differently, like Toyota's Agile production method that prompted software engineering teams to think differently about the software development lifecycle.

In the best cases, we gain a competitive advantage by thinking differently. According to NYU professor Adam Brandenburger, many innovative businesses were successful because they looked at the world differently than most. They looked at what was right in front of them, but in a way that escapes most people.

I've spent the last few years helping companies build successful, lasting FinOps programs, and my work has prompted me to think differently about FinOps, looking at it in a way that escapes most people.

I'm going to share my journey and I hope that by the end of this post, you'll think differently about FinOps, too.

A FinOps by Any Other Name

When most people talk about FinOps, they talk about cost optimization. And that makes sense because most companies' FinOps journeys start with cost optimization.

Cloud spend simply isn't a priority until it's higher than you forecasted, and then you want to re-align it with your budget as soon as possible by cutting and optimizing costs wherever possible.

But I've seen many teams accomplish optimization quick-wins just to see their spend rise again 6 to 12 months later.

Or worse, I've seen teams choose not to do the work at all, which leaves your company's FinOps goals in a lurch.

Operate Differently

So, what happened? Why didn't cost optimization work?

Let's revisit your FinOps goals: Maybe you want to cut costs 20% to stay in line with your budget. Have your teams architected for cost? Cutting costs will require changing the way they build and run workloads.

Or, you want better visibility into which products cost the most to operate in the cloud. Have your teams tagged and separated workloads along product lines? Allocating spend accurately will require changing the tools and processes teams use.

This is where the change in perspective comes in: FinOps goals always require a component of cultural or procedural change, and your people and business processes won't change solely because you tell them to. People need to be incentivized to change. Business processes need to be easy to adopt.

To accomplish your FinOps goals, you need to change the way your business manages cloud spend.

Cost Management, Not Cost Optimization

The very definition of FinOps describes it as "an evolving cloud financial management discipline and cultural practice". Changing the way your business manages cloud spend requires discipline and cultural change.

Cost optimization didn't solve your FinOps challenges because it didn't account for the cultural component of your cloud spend. You need your people, processes, and technology aligned with your FinOps goals. If you want to learn more, check out my blog post and talk where I dive deep on this topic.

FinOps is not about optimizing spend one time. It's about managing spend long-term which can include, but is not limited to, cost optimization. This is the way to think differently to solve your FinOps challenges, and set your company up for success.

How Nature Ecosystems Made Me a Better Engineer

Jesse DeRose — Tue, 13 Dec 2022 00:23:17 +0000

In late 2019, I was in a rut professionally. I wanted to hone my business skills but kept hitting walls.

It wasn’t for lack of work. Every role provided opportunities but my team's primary responsibilities were technical so my team didn't understand why I cared about non-technical work. If I pushed, they'd simply say "that's the way it is."

I felt stuck. And it sucked.

On a whim, I joined two ex-colleagues in an exciting venture they assured me would allow me to practice both my technical and business skill sets.

Systems in Nature

My boss and I spent some time getting to know each other in our first one-on-one. I talked about my background in DevOps and she talked about her background in nature ecosystems.

That's right: nature ecosystems. She started her career studying how plant and animal populations in a given geographic region impact each other–like how a predator population thriving might cause a prey population to dwindle.

She noticed this relationship existed among all plants and animals in the region. Each group's behavior influenced other groups' behavior, and vice versa. They depended on each other for survival.

This is known as a system: a group of interdependent parts (the plants and animals) that form a complex, unified whole with a specific purpose (in this case, survival).

And she realized these systems exist in technology companies too–the only difference is the names of the interconnected parts.

As she talked, I had the following flashback.

Systems in Organizations

Years ago, I worked in an engineering team that managed a subset of my company's core product. We attempted to optimize all aspects of our microservice but could never quite get the results we wanted.

Eventually we realized our microservice itself wasn't the bottleneck. The places where our microservice interacted with other parts of the product were the bottleneck.

Weird intermittent errors? Happened when other teams deployed new builds that weren't compatible with our existing build.

Query response times not going down? The function included a call to a microservice outside our control.

And ironically, our microservice was causing the same problems for other teams.

There’s a joke about two fish swimming together. One fish says to the other "how's the water today?" and the other replies "what the hell is water?" The joke is that we're not always aware of our surroundings. We get so caught up in our routine, in our microcosm of focus, that we don’t know about "the big picture", the holistic system we're working within.

This is exactly what happened to my team: we didn't know about the water we were swimming in. We didn't know how our actions impacted others, and thus couldn't change how our actions impacted others.

Not Knowing is a Risk

This was my path forward. When my boss gave me visibility into systems, I realized my previous lack of visibility had caused my rut. I had inadvertently proposed work that required systemic visibility without understanding what systemic visibility was or why it was important. Subsequently, my team didn’t buy into any of it.

And I don't blame them. Throughout my career, nobody told me or my teams about the water we were swimming in, the system that connected us to other teams and required TLC to ensure our company's success. Existing sprint and roadmap planning processes were accepted as-is because "that's the way things are" rather than because that's the way we deliver maximum value to users.

Lack of discussion about the systems that make up our organizations is a business risk.

When a team doesn't know how its work impacts other parts of the business, it's far more likely to take actions that slow or prevent the company from achieving its goals. A platform team I worked in believed documenting our deployment processes was a waste of time. What they didn't realize was the target audience for our platform struggled to use it due to poor documentation. Without this documentation, deployment velocity decreased, and key feature releases and bug fixes were delayed.

And this risk isn't limited to technical teams. In a previous role, my Head of Marketing regularly asked my team to write technical content. The requests were rarely fulfilled because my team's primary focus was client projects. My team didn't realize less technical content decreased the success of marketing campaigns, and attracted fewer potential clients.

Ask Your Doctor if Systems Thinking is Right for You

Now when I join an engineering team, I immediately ask how my team fits into the organization. One of my favorite activities is comparing roadmaps to identify incongruences, competing priorities, and misaligned incentive structures.

This may seem like a business skill that only project managers or product owners should care about but understanding how one's actions impact other parts of the organization is equally important for engineers. It should be part of onboarding for every employee. Long gone is the information silo era (hopefully). We need to educate employees on systems thinking for the business context era.

If you aren't sure how your team's work impacts other teams in the organization, ask. Normalize systems thinking–it's not just about OKRs and business strategy. Be the change you want to see.

And if you know how your team's work impacts other teams in the organization, you should still ask. Your mental model may not line up with others'.

The more visibility we have into how we work together, the better we can work together. Simon Sinek once said "people don’t buy what you do, they buy why you do it." Systems thinking helps me understand the "why" behind each team I work with so I know how to help them and our company succeed.

Cover image by Brett Sayles.

Controlling Cloud Costs with Culture

Jesse DeRose — Tue, 25 Oct 2022 23:44:25 +0000

As companies move more workloads into public cloud platforms like AWS, GCP, and Azure, managing those costs becomes more important and more complex. Even when armed with Cloud Financial Management (CFM) frameworks and best practices, a lot of companies still struggle to manage their cloud spend. To succeed long-term, companies also need to focus on the people involved, creating a cost-conscious culture that ensures employees are engaged and on board.

In this article, we'll dive into why frameworks and best practices aren't enough and three key steps to create a cost-conscious culture that will lead your company into its CFM golden age.

Why frameworks and best practices aren't enough

I've spent the last three years helping organizations succeed in their CFM journeys, and I noticed a trend. We'll take one client as an example.

The organization's CFM journey started when someone in senior leadership noticed cloud spend was "too high". Maybe it spiked astronomically from last month, or maybe it was growing slowly month over month, year over year. To solve this problem, the VP of Engineering decided to cut spend 20% over the next 6 months.

Extensive research supplied reactive, short-term suggestions like purchasing reservations or turning off unused resources. The VP dropped tickets into their teams' queues, those teams performed the work in upcoming sprints, and cloud spend dropped. Fantastic.

With the tickets completed, teams returned to their regularly scheduled backlog. And that's when it happened–the org's cloud spend slowly climbed up and up again, and the organization was back where it started. (The Duckbill Group refers to this as the cloud cost cycle of pain.)

I saw this trend in organizations of all shapes and sizes: employees completed the CFM work that was asked of them and then went back to their day job. And that's no fault of their own–they were hired to build and maintain their company's rock-star product. They're incentivized to focus on shipping code.

This is why CFM frameworks and business practices aren't enough–by themselves, they're not part of an organization's "business as usual". Employees won't necessarily embody these ideas on their own. To change that, you need technical solutions and people solutions. For a CFM practice to be successful, to ensure "business as usual" includes CFM work, you need a cost-conscious culture.

Do I really have a culture problem?

I know, I know–you're probably thinking "I don't have a culture problem."

Well.

In the 2022 State of FinOps Survey, 30% of all respondents said making engineers take action was the biggest challenge they faced.

An additional 22% reported struggles with organizational adoption of FinOps.

That's 52% of respondents who said their main challenge in implementing CFM practices was related to culture. That's right, over half of respondents expressed culture change as a major hurdle for successful CFM practices.

If you're struggling to keep your cloud spend under control, you may have a culture problem. And you're not alone–lots of companies are experiencing the same thing.

I get it–culture change is hard. Nobody wants to admit they have a culture problem and even if they do, finding a solution is challenging. But fear not! The organizations who built successful CFM practices shared three key business practices that you can use to create your own cost-conscious culture.

Practice 1: Leadership Buy-In and Communication

Leadership must buy-in to the importance of CFM and communicate its importance to all key players in the company. There are two major questions to think about here:

Why does the company need to care about CFM?

If you run your company's workloads in data centers, that spend is usually classified as capital expenditure (CapEx): upfront investments, depreciated over time. But when you run workloads in the public cloud, that spend is usually classified as operational expenditure (OpEx): variable spend accrued per minute that varies day-to-day and month-to-month. Public cloud spend can cause major spikes in ways data center spend never could. Yikes.

CFM practices create visibility into workload spend on your public cloud platforms. Think about CFM work in terms of business value: how is your public cloud spend split amongst your products and services? Is spend split evenly across all your products? Is it heavily focused in one product? Do profits for that product outweigh its cloud spend? Your bill won't give you any of these insights unless you implement CFM practices like user-defined cost allocation tags and well-labeled linked accounts.

Similarly, how is cloud spend influenced by business KPIs? If you're a SaaS company, for example, you might care about business growth as a function of the number of customers on your platform. So how much money are you spending on the public cloud for every customer you have? What's the cost of keeping each customer on your platform? Is that more than you're charging them to use the platform? CFM will help you answer those questions, and your business leaders need to set goals accordingly to prioritize that work.

Note that I say leaders and not managers–the two aren't necessarily the same. Key influential leaders can come from any level of an organization. Think about the people who are passionate and vocal about their work–those are the people you want bought-in on CFM as early as possible because they can help you champion it to other parts of the organization.

How does this goal fit in versus other priorities?

Also note that I say leaders broadly, not just Engineering leaders. Offering a technology-based product requires input from multiple stakeholders like Engineering, IT, Security, Product, and Design.

Leadership needs to communicate the importance of CFM work to all of those stakeholders, and ensure everyone's priorities are aligned. If stakeholders don't agree where CFM fits versus other organizational priorities, they might send mixed signals to Engineering teams. And we've all been on a team where priorities change week to week or even day to day. Nothing gets done because you're constantly shifting focus, the work that does get finished is lower quality, and your chances for burn-out increase. Nobody wants that.

Similarly, these stakeholders may have vital business context that influences Engineering teams' ability to optimize costs. For example, I worked with one team who identified terabytes of infrequently-accessed customer data that could move into a cheaper storage tier. This particular storage tier was cheaper because data in this tier was retrieved in a number of hours rather than a number of seconds. However, when the team discussed this change during a meeting, a Customer Success team member reminded them of a service-level agreement that required all customer data be readily accessible within minutes. This meant the team couldn't move the data without violating the SLA, and thus couldn't optimize that spend without changing the SLA.

Practice 2: Incentivize Employees to Care

When leadership communicates the importance of CFM work, they're telling employees where to focus their time. But for teams to accept this new way of working, they need to buy into it. And to get buy-in from employees, you need to incentivize them to change.

How each individual's incentives align with business goals will vary. Every person has different incentives so ask them what they care about. Make them feel heard. If you're their manager, bring it up in your next one-on-one. Some key topics to consider:

What's in it for the employee? Is this an opportunity or a threat? Is your organization's CFM program going to improve their quality of work life or be detrimental to it?
What's your organization's track record with organizational change? If your organization has successfully implemented org-wide changes in the past, employees are more likely to believe your CFM program will also succeed, and subsequently buy in.
What other factors are impacting the employee outside of this change? Maybe this person is experiencing instability in their team or home life and fear the new habits you're asking them to embody will create additional instability.
What are the employee's personal values and motivators? Most of us work for money but is that our primary, driving incentive? The Great Resignation and Quiet Quitting suggest other factors might incentivize employees to show up to work every day.

Once you know what your employees care about, you need to help them build new, proactive, cost-conscious habits that support your CFM goals. Charles Duhigg's The Power of Habits can provide additional guidance here.

Practice 3: Make the Happy Path the Easy Path

"The Happy Path" is a concept commonly used in product development to describe the default scenario a user experiences to complete a task with no exceptional or error conditions. Think of it as what you expect to happen when you checkout on Amazon.com, or the default process your Engineering teams follow to deploy a new feature into production.

To build a cost-conscious culture, you need to embed proactive cost habits into your happy path workflows, into the default ways your teams work. You need to embed proactive cost habits into the design, strategy, and provisioning processes for your engineering teams. If Engineering teams aren't thinking about cloud cost in their daily work, they're not incorporating it into their daily work, and you'll end up with reactive, short-term work all over again.

For example, your engineering team already has a happy path for deploying microservices to the cloud, likely involving a CI/CD pipeline and a documented business process. An easy way to embed proactive cost habits into this workflow is to add cost allocation tags to resources as they move through this pipeline. You could even add the tags to your Infrastructure as Code files so every resource deployed through the pipeline automatically receives tags, and is clearly allocated to a specific team or department.

Your engineering team also already has a happy path for managing development cycles, via a number of structured meetings like standups, sprint groomings, and retros. Use these meetings to proactively look at historical cloud spend or review optimization opportunities. Teams could even discuss cost as part of the architecture design process–does your new microservice really need 16 vCPU and 64 GB RAM at all times? Or can you autoscale smaller compute resources as demand fluctuates?

These changes require a little engineering effort upfront but ultimately allow you to gather the cloud cost data you need and teams to stay focused on what they do best. It's a win-win.

For teams to embrace proactive, cost-conscious habits, they need to be surrounded by tools and processes that incentivize those habits. That's the happy path idea. But not every tool and process in your organization is designed to incentivize those new habits. A lot of them might actually incentivize teams to revert to old habits that are counterproductive to your CFM goals.

This leads to the easy path portion of our discussion. Have you ever tried to complete a task and found the default workflow convoluted, frustrating, or counterproductive? I've seen many teams reinvent the wheel simply because the default workflow wasn't easy or aligned with business goals. Empower teams to change tools and processes that don't foster cost-conscious habits. If employees don't shift away from old habits and behaviors, they won't embody cost-conscious habits and behaviors, and your CFM program won't be successful.

It's Dangerous to Go Alone

Two final thoughts: first, there will be disagreements along the way. That's okay. In fact, that's healthy. When you bring different perspectives together, they're not all going to agree. What is important is that everyone feels heard and moves forward together.

Second, culture change takes work. Yes, lots of hard work. This is not something you will implement overnight. It’s not even something you’ll see dividends in immediately. Define how you plan to measure success early, and measure often. For example, one team I worked with regularly polled employees to rate their agreement or disagreement with various statements about cost-conscious habits. Over many months of CFM work to create a cost-conscious culture, most employees reported increased alignment with, and embodiment of, cost-conscious habits.

And I get it–culture change is hard. Nobody wants to admit they have a culture problem and even if they do, finding a solution is challenging. Changing a group’s culture requires a lot of soft skill work, and that’s hard to define in terms of ROI and business KPIs so any culture change projects generally don’t happen at best or are doomed to fail with no clear path to outcomes at worst.

Focus on getting senior leadership buy-in, incentivizing your employees to care about cloud costs, and leveraging your existing tools and processes.

Once you’ve got a cost-conscious culture, then you can introduce CFM frameworks and best practices. Shout out to the FinOps Foundation which has done an amazing job centralizing and highlighting helpful resources.

And if you get stuck–no worries! Reach out to me. I’m happy to offer assistance.

Do I Really Need to Travel for Consulting Work?

Jesse DeRose — Wed, 21 Sep 2022 17:18:10 +0000

About two months ago I applied for a consultant role at a large technology consulting firm. A recruiter responded with:

I can see you in the system but you marked you’re not able to travel - is that still the case?

I had listed a preference for zero travel. There's still a pandemic and I live with someone at higher risk for infections. But, as with any job application, there's always room for negotiation.

Their next comment surprised me even more:

Each project has different client needs so right now we are connecting with candidates that are open to up to 50% travel just to cover any potential client asks. Face to face engagement is a major part of the 'consulting' world so to cover all bases 50% is the benchmark at this time.

50% Travel? In This Economy?

In 2015, I worked for a technology consulting firm that sent all of its consultants on-site to client locations. Teams rarely hired remote employees and when they did, those employees were viewed as "the black sheep"–important to keep in the loop but not part of ad-hoc conversations happening in the office.

It made sense for consultants to be on-site because employees were expected to be on-site.

Then COVID-19 became a major health risk in 2020, the US Federal government instituted a stay-at-home order, and remote work became A Thing^TM. Tons of companies migrated to a temporary work-from-home model until "the coast was clear". Many permanently instituted remote policies, allowing employees to work from anywhere in the country as long as they still joined team meetings and finished tasks on time.

Researchers proved remote work works and is desirable among employees, despite companies' push to "get back to normal" with employees in the office.

So why do consultants need to be on-site for a project when teams are already holding stand-ups and meetings online? Or, put a different way, what's the benefit of a consultant being on-site, in-person?

Building Trust

One major component of high-performing teams is trust, which is most easily built in-person. This means any consultant working a new engagement is automatically at a disadvantage because they have to project expertise while also building rapport and learning about the client's key players, culture, and communication styles.

Trust requires building individual relationships, incrementally, over time. Historically, a consultant accomplished this goal in person: traveling on-site to client offices, sitting in the same space as client teams, and participating in all team meetings and conversations. But over the last two decades, researchers proved groups working remotely can build as much trust as groups working in person and many teams adopted digital solutions that allow employees to remotely build trust in the same ways they previously built trust in person. Trust is still built faster in person but can now be built just as strong remotely.

There are other benefits to working on-site with a team but arguably all of a consultant's work could be done remotely.

Interviewing stakeholders and building trust? Video calls and asynchronous messaging tools.

Reviewing documentation and processes? Web-based wikis or project management software.

Collaborating with teams? Real-time productivity suites.

So, if consultants can do all this remotely, why would a firm still require them to travel on-site for a client engagement?

Potential Client Asks

Consultants don't need to be on-site for most engagements but recruiting firms like this one require travel "just in case". This firm would send consultants on-site up to 50% of the time simply because the client asked for it, even if the client didn't actually need it.

In many cases, the client believes they need a consultant on-site because they haven't updated their organizational models for the remote age. Even when flights and hotel prices are incredibly cost prohibitive or prices surge due to large events happening in the area, many companies still insist on bearing the financial burden simply to maintain in-person stand-up meetings for all employees.

The consulting firm isn't to blame here but is adding to the problem. They're broadly greenlighting travel without clear guidelines for acceptable, reasonable client requests. This behavior incentivizes the client to continue operating with an outdated organizational model which ultimately impacts the consultant more than either company involved.

Not All Roles are Like This

Not every consulting firm requires travel for client engagements. I've run successful engagements in a fully remote environment and clients didn't complain once.

When I brought this topic up with some colleagues, one of them said "the nature of consultative work is bowing to the wishes of the clients." I agree with the sentiment but when a recruiter tells me their firm requires up to 50% travel "just in case the client says so", I have a lot of questions about how that firm operates and where their loyalties lie. To me, it sounds like that firm's executives don't have their employees' best interests at heart–they're willing to send them out in risk of COVID-19, monkeypox, and any other number of health and safety issues because that's the working model they and their clients are used to and they just want this whole thing to go "back to normal".

I hate to tell you but the new normal won't look like the old normal. Office work has changed dramatically and companies need to reflect on how their organization will integrate that change. Remote work isn't perfect but companies need to adopt it as a major component of technology-focused roles, consultative or not. Companies need to keep their employees' health and happiness in mind when considering travel requirements. In-person work can still happen but should have an intentional reason for happening.

Cover photo: Oleksandr Pidvalnyi

The Art of Storytelling in Engineering

Jesse DeRose — Thu, 18 Aug 2022 22:54:00 +0000

The last few years have forced companies and employees to rethink how remote work is performed. One of the most critical components of a successful remote workforce is communication. Communication is especially important for Engineering roles like SREs and DevOps that require collaborating with multiple disciplines across an organization.

Different teams bring different function-specific language, business context, and team-specific goals to every discussion. How can teams effectively communicate all of that critical knowledge with each other? With storytelling.

Every employee in a company views their work differently, much in the same way every character in a story views the world differently. Collaboratively building shared mental models of the problem at hand allows teams to identify and address those different views to form a more resilient solution.

Similarly, every employee is driven by their own professional agenda, much in the same way every character in a story is driven by their own personal agenda. Identifying all involved employees’ wants and addressing them in the process of building a solution increases employee engagement and buy-in for the solution, which ultimately leads to better engagement with the solution.

Storytelling isn’t always about what’s said, but what’s conveyed. Non-verbal communication like body language, posture, and tone of voice all influence the delivery and understanding of information. Employees will pick up on these cues and tune in or tune out of the conversation accordingly, so engaging your teammates with verbal and non-verbal communication is critical for effective communication.

Finally, let your team be the hero of their own story. Give your team a clear problem statement, position your company’s values as guardrails for solving the problem, and let the team find the solution themselves.

Want to hear more? Check out my website or shoot me an email!