DEV Community: Michael Zion

Why & When to use GitOps: Use-Cases & Principles

Michael Zion — Tue, 11 Mar 2025 21:43:41 +0000

Optimizing Kubernetes Management with GitOps and CD Tools.

We’ve come a long way from the days of deploying releases to individual machines, whether manually or by an automated process.

The de-facto standard today is to deploy and manage the application stack using an orchestrator, usually Kubernetes.

However, Kubernetes carries its own challenges. Although you can manage your stack in one centralized place, you still have to manage deployments, updates, rollbacks, and the overall infra.

When Kubernetes was in its early years, services were deployed either manually or through CI/CD using kubectl or Helm.

There was nothing to describe the desired state and no easy way to keep track of changes or revert them. To solve these issues, GitOps was added to the mix.

Single Source of Truth with Increased Visibility and Operability

The idea behind GitOps is a git repo that describes the desired state of the infrastructure and the application stack that runs on it.

Combined with CI/CD, GitOps allows for fast and efficient infra provisioning and application lifecycle management.

In the context of GitOps and Kubernetes, any change to the source code triggers a change in the cluster or application stack.

Changes are tracked and releases are tagged with a version and can be rolled back as fast as they can be deployed.

Best of all, no one is running manual commands which can be very error prone. In essence, GitOps centralises all operations related to deployments and infra management while allowing various teams to both collaborate and work independently.

Continuous Delivery Tools to Enhance GitOps

However, GitOps and CI/CD for Kubernetes still lack something in terms of really being able to manage the infra and application stack.

Managing deployments and infra becomes easier with GitOps thanks to declarative continuous delivery tools for Kubernetes such as Argo CD or Flux CD.

These tools act as a wrapper around your codebase which translates everything declarative into actual state. Your codebase holds all of your Kubernetes manifests, Helm charts, and configuration files. Argo CD then takes it, evaluates it, and applies it to the cluster.

Such tools also offer intuitive and convenient UI to visualize and manage everything that is going on in the cluster.

It’s good to be able to revert a deployment or fix a configuration issue through the UI and on the spot rather than going through the process of committing code and waiting for it to sync. This is especially true in case of an emergency or when developing and debugging.

The Real Benefits of Combining GitOps and CD Tools

When put into the context of everyday work of developers and DevOps, the real benefits of GitOps and CD tools come into light. What GitOps and CD tools offer is a nice package of intuitive and secure easy-to-use framework for management and collaboration.

Ease of Use and Visibility into the State of the Application and Cluster

The truth is that although not too difficult, setting up a GitOps codebase with CD tools can be a complex task. However, once in place, they grant ease of use with increased security.

Reading Kubernetes manifests to understand the complex relationship of services and components is not easy. It’s even harder to debug and troubleshoot issues with no centralized control dashboard.

It’s easier with CD tools because they visualize everything that the code represents.
It’s easier to discern the components of a service or the various CRDs that have been added to the cluster when they appear in front of you with all their dependencies and associations.

CD tools have robust control, logging, and debugging features which allow developers and DevOps to quickly assess the status of the cluster and respond accordingly. There’s no need to run kubectl commands or tail pod logs. Everything is intuitively visualized in the CD tool.

Secure Operations

Having everything visualised and controllable from outside the cluster, does away with the need to access the cluster itself.

‍Developers and DevOps can be given roles through RBAC to determine which resources they have access to and what they can do in the CD tool itself rather than in the cluster.

Although access is ultimately given to components of the cluster and application, it’s still better than having to manage service accounts and having dozens of developers running exec into pods.

For additional security, CD tools sit inside the cluster and communicate with native API so their function takes place inside the cluster and not outside. There’s no need to open the cluster to external CI/CD pipelines.

Quick Recovery, Consistency, and Guardrails

Recovery is built into CD Tools. CD tools continuously monitor the state of the cluster and compare with what the code says.

In case of a mismatch, the CD tool attempts to reconcile between the two. If it fails, it will retry several times before sending an alert and even pausing sync.

Retries are especially important in complex systems with vast networks of dependencies between services and components.

Even if the code dictates service dependencies and the order of deployment, dependencies could still fail.

By retry, CD tools give all components of the system more time and more chances to go online before failing the deployment and pausing progress.

Since there are no imperative commands running without a state, GitOps prevents the occurrence of hanging resources. If you remove a manifest file or parts of it, the CD tools will sync immediately, without any CI/CD pipeline, and remove the corresponding resources and components.

Management of Multiple Environments and Applications

With inherent security, ease of use, and built-in recovery, GitOps and CD tools are easily expanded to support the configuration and management of multi-env, multi-app stacks. It becomes much easier to separate environments:

Different environments for development and production.
Regional stacks for compliance or disaster recovery.
Separate applications for business units in the organization.

Since the initial setup is most of what it takes to get GitOps and CD tools going, it’s incredibly easy to set up multiple stacks and manage both separately, as they are different stacks, and side-by-side in one place, as they are centrally managed.

The Application Needs GitOps and CD Tools

Keeping the application stack and cluster in order and in the desired state is the main function of GitOps and CD tools. There's no guessing how the cluster should be or what should be in and who or what put it there. Everything is conveniently visualized with a vast array of actions that can be performed right from within the UI. GitOps and CD tools give developers and DevOps the ability to work independently in a secure and safe way. Secure in terms of access that is restricted to the CD tools itself. Safe in terms of the ability to quickly revert changes and the self-healing properties of CD tools. It’s not the easiest task to set up GitOps with CD tools, but the agility, convenience, productivity and security that they offer allow for well-managed operations. Well-managed operations then allow for faster and safer development pipelines to further advance your application and product.

How to build a DevOps team

Michael Zion — Tue, 01 Oct 2024 22:27:55 +0000

If you're considering how to build a DevOps team the best way possible, this one's for you.

This blog post is the result of my advice for people building DevOps teams: CTOs, VPs of R&D, Team Leaders, and more.

TL;DR

I'll give you the bottom line.
These are the steps for building your DevOps team:

Polish your DevOps Philosophy
Understand the DevOps Lead responsibilities: Product, Project, People, Service, Architecture
Define your Team's Mission Statement
Set useful DevOps Goals & Practices
Set Guiding DevOps Principles
Define the DevOps team's roadmap & strategy
Set a definition of done
Common DevOps teams pitfalls
Calculate your team's required capacity
Hire the right DevOps Engineers

You can also apply these to existing DevOps teams and level-up your organization.
Let's get started!

Polish your DevOps Philosophy

You must first understand why organizations need DevOps, and how it can work in practice.

DevOps is an enabler role.

It's meant to enable the developers to build, improve, and take ownership over the system.

Let's break it down:

Build = Create something new. Could be a new microservice, a new database, or a new monitoring dashboard.
Improve = Introduce a change into something that exists. Could be fixing a bug, changing a database schema, or changing an alert threshold.
Take ownership = Take charge of problems that arise with what you built and improved. This means when something needs improvement, the owner is the one who does it.

The above are what the developers should do.

Now, DevOps should enable it.

Note, to enable is not to give full permissions and let them have fun.

To enable is to give developers what they need to build, improve, and own the system.

But, do it in a way that focuses the developers' energy in the right direction, and in a safe way.

Understand the DevOps Lead Responsibilities

You have 5 hats as a DevOps Lead:

Product (Platform)
Your clients are your company's developers.
Provide them with tools, knowledge, and automations.
Tools = Polished automations.
Automations = Automated knowledge.
Knowledge = Hard-earned information & insights.
Understand their requirements, and use Tools, Automations, and Knowledge to fulfill them.
Understand why they want what they want: Is it because of an underlying issue with the system? If yes, solve it before building anything.
If a tool or automation will save time, consider building or implementing it.
Service
Don't let your developers wait until you do something "the right way".
Sometimes they need immediate help to complete something.
Help them first, and invest time later in automation and tooling.
Project
Managing your DevOps team requires managing the work it does.
Turn the team's philosophy into a mission.
Turn the team's mission into goals.
Turn the team's goals into a roadmap.
Turn the team's roadmap into tasks.
Prioritize the tasks.
Set simple roles and responsibilities.
Hold your team members accountable to progression.
Make it easy to inform relevant team members on updates.
Make it easy to consult with a teammate on a subject of their expertise.
People
Each DevOps team member has type of work and tasks they enjoy more.
One enjoys sharing knowledge, another enjoys being of service, and some enjoy building tools.
Different teammates are also interested in different subjects: Monitoring, Infrastructure, CI/CD, etc.
A team member that's working on what they enjoy is more fulfilled and productive.
Strive to overlap each teammate's goals with the company's goals.
Architecture
You have 2 architectures to worry about.
1) The company's product architecture (built by the developers).
2) The DevOps platform's architecture (built by the DevOps team).
Help the developers understand and control the application's effect on the infrastructure.
Enable the developers to make informed decisions by providing context.
Finally, build the platform to support the developers requirements.
At every step, only limit decisions that damage the company.
Examples: High cost, Stability impairment, Restricts observability, etc.

Define Your DevOps Team's Mission Statement

Here's one for you:

"Enable the developers to build, improve, and own the system".

It's pretty minimal, so it's going to help your team stay focused.

A healthy sign it catches on: When the team debates a decision regarding a task, they ask "Does it help us enable the developers to build, improve, or own the system?".

That's when you win.

Set Useful DevOps Goals & Practices

Your company's success will be determined by its speed and its product's quality.

To make it happen, you should know programming is science, not maths - I'll explain.

People used to think programming will be a mathematical discipline.
They thought programmers will write functions and mathematically prove them.

Not what happened - Programming is a scientific discipline.

You write code, your test it, and you assume it's good - until one test fails.

In essence, you're experimenting.

You might ask: "wtf? how's this related to setting useful goals?"

The answer is that the first thing you want to enable is running experiments easily.

So here are some useful goals:

Developers can easily test their code in a consistent manner
Production and testing environments are identical (Production will benefit from the quality of the tests)
Developers can easily collaborate
Developers can understand the state of the system and the impact of changes

Let's translate those goals into smaller goals or practices that will achieve the goals:

Developers can create a testing/production environment with "One-Click"
There's a continuous integration process that enables the developers to collaborate by agreeing on the current up-to-date version of the system
Auto-create dashboards and alerts for new services

Set Guiding DevOps Principles

Some useful DevOps principles that help save your team time, improve the speed of delivery, and keep the system healthy:

PoC before doing things "the right way"
Make it work, then make it better
The entire system should be fully recoverable from Git
Use tools with a big community and well-documented interface
Equip key-developers with DevOps knowledge to be the first point-of-contact for their team members (super users)

Define the DevOps Team's Roadmap & Strategy

Roadmap = Goals * Strategy.

Once you set the goals (as mentioned above), you can start prioritizing.

♯1 - The DevOps Categories

Enabling developers requires a DevOps Engineer to handle and enable the following:

Provision infrastructure
Deploy workloads
Monitor the system
Recover from issues
Scale up and down
Track & test changes (Codebase Management)
Secure the system
Store & retrieve data
Configure the system

♯2 - Examine each goal through each category

Every DevOps goal you set should be examined through the lens of each category.

The reason is that together the categories cover each aspect of the building, improving, and owning of a software operation.

Let's do an example:

Goal: Create a "One-Click Environment Automation"
Categories to address:
- How should its infrastructure be provisioned?
- How should its workloads be deployed?
- How should its metrics and logs be sent, stored, and queried?
- ...

♯3 - Strategic Principles

Reach at least 50% capacity of working on the DevOps goals as soon as you responsibly can - Support the developers and teach them how to self-support to achieve that
Easy to modify > Perfect - When you do something that isn't perfect due to a lack of time, do it in such a way that modifying it later on to improve it is easy
Prerequisites first: Codebase Management -> Infrastructure -> Deployment -> Configuration -> Data Management.
At least moderate foundations quickly, reinforce later, but never weak foundations: If you don't use any boilerplates, or if you are not proficient in something early on, and it's a foundation (like infrastructure), then don't give up and build weak foundations, but also don't over-invest and build strong ones if it's too time consuming. Instead, moderately invest in the foundation, and revisit it later.

Set a Definition of Done

Also known as a Definition of done.

Ask the following questions for every single component in your system:

Monitoring: Are there metrics, logs, traces, and alerts setup in an actionable way?
Availability: Is there a mechanism to keep it alive during incidents?
Resiliency: Can it recover from an error quickly?
Recovery: Can it be fully restored to a previous state?
Testability: Is it possible to test changes to it?
Deliverability: Is there a process to release changes to it?
Persistency: Will its data persist if the system is hindered?
Integrable: Does it have a consistent and predictable interface allowing integration with it?
Security: Is it accessible only by the parts of the system that absolutely need it?
Dependencies: Are its dependencies fully tracked and managed?

Common DevOps Team Pitfalls

Pitfall: DevOps work blocks developers work
Indicator: Developers need to wait for DevOps team changes to complete before continuing work
Cause: The DevOps team doesn't utilize its own practices (AKA shoemaker's son always goes barefoot)
Pitfall: Only support developers and maintain the system
Indicator: No progress on any DevOps goal or task
Cause: Either there are no clear DevOps goals or repeating developers requests haven't been automated
Pitfall: Adopting 'Best' practices instead of 'Suitable' practices
Indicator: Introducing methodologies and adhering to principles that go against the company's goals
Cause: Prioritizing methodology over company goals, usually because of a disconnect from the company goals or due to lack of DevOps experience

Calculate Your DevOps Team's Required Capacity

Required DevOps Capacity = (Scale * Complexity) / Leverage.

Leverage = Level of DevOps Engineers * Company Resources * Team Focus
Scale = Number of instances of each component * Number of people in the organization
Complexity = Number of components * Number of teams

Hire the Right DevOps Engineers

The types of DevOps Engineers:

Barrels vs. Ammo
~ Ammo = People who can complete tasks but won't initiate them.
~ Barrel = People who understand what tasks are needed next but won't complete them.
Interviewer Tip: During the technical interview with a candidate ask about past DevOps accomplishments, and ask how the tasks were created and who did them.
Aspiration-Oriented vs. Prevention-Oriented
~ Aspiration-Oriented = Has goals, positive feedback encourages and focuses them while negative feedback discourages them and kicks them off track.
~ Prevention-Oriented = Avoids problems, positive feedback makes them lay back and reduce capacity while negative feedback focuses them and keeps them on track.
Interviewer tip: See if there's a common theme for projects the candidate did in the past. People focused on security tasks and deep attention to specific details are more likely to be prevention-oriented, while people who initiated many projects spanning multiple (DevOps) categories are more likely to be aspiration-oriented.

Working with service providers:

Set clear desired results
And let the DevOps service provider assist in exploring the goal and provide perspective from other companies.
Expect transparency on progress
And judge it against the latest plan.
Expect clear planning
And make sure it has clear goals, takes into consideration the risks, and has a strategy that adheres to your DevOps principles.

Summary

Lots of stuff covered over this one-pager, and still much was left outside.

If you take away 1 thing from it, let it be this: A simple DevOps team mission statement is the most significant thing you need.

It sounds over-simplistic and abstract, but without it there is not guiding principle for how DevOps in the organization should look like.

Hope you enjoyed, and send me an email at michael@meteorops.com if there's anything else you'd like to see in here.

Don't waste 6 months on hiring DevOps, take the 10 Hours DevOps Pill

Michael Zion — Thu, 04 Jan 2024 14:55:06 +0000

A practical playbook for founders and engineering managers to stop wasting time trying to get started with DevOps the right way

Is this article for you?

Maybe you're a CTO.
Maybe you're an engineering manager.
If you think about hiring you first DevOps Engineer - this article is for you.
It will save you 6 months of wasted mental capacity.

How? By dedicating 10 hours to planning.
I'll share with you here exactly what to do during those 10 hours.

You might know this story TOO well

~ "I need to focus on the product"

You are building your company’s product.
You start thinking about the development process and building or improving production.
Then, you read about DevOps somewhere and think to yourself: “This seems like what I need”.

~ "DevOps is expensive"

You start looking for a DevOps Engineer, but you find out the standard DevOps salary - “Oh, that’s pretty expensive for our budget”.
So you decide to wait. “Hiring a developer is more important right now” you tell yourself.

A month goes by, and you and your team are grinding away, building a kickass product.
But alas! The missing development automation hurts collaboration, the cloud infrastructure is a collection of manual changes - and you’re not sure if the system is built well.

So, you make a decision, and you post your first “DevOps Position”, congrats!

~ "I'll interview some candidates"

Candidates start streaming in, and you start interviewing.
After interviewing the 3rd candidate, you think to yourself “this is not it”.
Every candidate is missing a different piece of the required knowledge.
One doesn’t know AWS, the other isn’t familiar with "", and the last one only worked with on-prem before.

And so, you re-evaluate.

~ "I'll do it myself"

The realization is this: You can’t hire on knowledge for this position, you have to hire on skill and character.
BUT, you feel like the knowledge is crucial, at least at this point.
So you make another decision - I’ll invest some more time in this “DevOps” thing - I’ll learn.
You watch DevOps tutorials on Youtube.
You even subscribe to various DevOps Newsletters, something you didn’t do before.
Another month goes by.

~ "What do other startups do?"

After learning and implementing DevOps bits and pieces all over the place, it doesn’t feel right.
“Am I re-inventing the wheel and complicating the future?” you might ask yourself.
You take a step back, zoom-out, and ask other startup founders what they did at this stage.

That’s when you find out you’re not alone - “Oh! So it’s tough for other founders as well!”
But, each founder tells you something different.
One founder tells you to hire a freelancer - but that seems like just another hiring model and doesn’t solve the problem.
Another founder tells you to just hire a full-time DevOps Engineer - but your resources are limited, and you’d rather hire more developers to build the product.
Until one founder tells you to hire an agency: “Instead of a freelancer, get a company that’s built to get your DevOps right”.

~ "Maybe an agency?"

You’re convinced, and you start looking for agencies.
You get recommendations, you search online, and you start booking calls.

*1st call with a DevOps agency
*“Hi there! Yes we’re the solution for you! Let’s get started!”

2nd call with a DevOps agency
“Yes! We’ll provide you with a strong DevOps Engineer! Let’s get started!”

~ "I'll do it myself"

You choose not to go on the 3rd call and you pause the process.

Why? Because you realize: “These guys didn’t remove any uncertainty for me - how are they different from a freelancer?”
So you make another decision.
You’ll continue to do DevOps yourself, and maybe delegate some tasks to other team members.

That’s when you find out it was a trap! You got yourself into a loop!
‍

~ "Oh no! it's a loop!"

After just one month you realize it’s not sustainable, and you repeat the same process:

Open DevOps position
Realize each candidate is missing something else
Fear for the budget
Stop the hiring
Complete DevOps tasks yourself
Consider a freelancer
Drop the idea because it's similar hiring
Consider an agency
Interview a bunch of agencies
No extra certainty or confidence from an agency
Do DevOps work yourself
Repeat

This loop is sometimes 2 months long, and sometimes 12 months long.

Either way, it slows down your company!
It occupies your thoughts.
It takes valuable time from you.
It speeds up global warming.

Maybe the global warming part was an exaggeration.

It’s amazing how many founders went through this exact same process.

Break the loop

I’m here to tell you:

You can break the DevOps hiring loop
You can promote DevOps efforts quickly AND accurately
You can save 6-12 months of stumbling around

You need just 10 hours with a DevOps Expert

You need just 10 hours with a DevOps expert.
What you need most is perspective from an experienced individual.
You don’t yet have the DevOps perspective or experience? No problem.
Get a DevOps Expert to consult you on a hourly basis.
I call it "The 10 Hours DevOps Pill" 💊

Requirements for the DevOps expert

Not anyone will fit as your trusted DevOps expert.

Use someone that answers these criteria:

Helped 10+ companies with DevOps - Has a wide perspective
Consulted CTOs and VPs of R&D before - Understands the weight of culture in adopting DevOps
Consulted DevOps Group and Team Leaders before - Understands how to manage the work
Did lots of DevOps hands-on work before - Understands how to do the work
You understand them - They know where you stand and have simple explanations. (Run away from the buzzwords throwers)

Spoiler: The contents of the 10 hours

2h - Share everything about your startup
2h - Ask what’s possible + What others did
2h - Set DevOps goals
2h - Choose a strategy
2h - Create a DevOps roadmap Let’s dive into the playbook for each meeting!

1st + 2nd Consultation Hours: Share, share, share

Share the state of your company with the DevOps expert.
Share your infrastructure, your code, your work process, everything!
Don’t be shy, and don’t justify - simply explain your startup’s goals and your reasoning.
(Sign an NDA first, of course)

3rd + 4th Consultation Hours: Understand what’s possible + Gain perspective

Take notes or record this call!

Ask the DevOps expert: What’s possible?

How fast can changes reach production?
How fast can you create new environments?
How easily can your developers collaborate?
How confidently can you change the system?
What uptime can the system have?
How aware can your team be of the system’s state?

Ask the DevOps expert: What did other startups do at this stage?

What goals did they have?
What DevOps efforts did they promote?
Why did they decide those DevOps efforts?
Which startups performed best? Why?
Are there industry-unique considerations?

5th + 6th Consultation Hours: Set DevOps goals

You now know what’s possible, and what other companies do.
Time to set short-term, mid-term, and long term goals.
If you want to dive deep into useful DevOps principles, check out the CTO DevOps Handbook here 👈🏼.

7th + 8th Consultation Hours: Strategize

Based on your goals, strengths, and restrictions, choose a strategy.
Discuss the following options with your DevOps expert.

Possible Strengths

Developers with DevOps background
High budget
Available time to manage the DevOps efforts

Possible Restrictions

No budget
Low budget
DevOps needs are temporary for now
No time to manage the DevOps efforts

	DevOps Engineer	Freelancer	Agency	Winner
Budget	High	Medium	Medium	Freelancer / Agency
Management Effort Required	High	High	Low-Medium	Agency
Hourly Flexibility	Low	Medium	High	Agency
Ease of bridging knowledge gaps	Low-Medium	Low-Medium	Medium-High	Agency
Ease of building trust	High	Medium-High	Low-Medium	DevOps Engineer

The available strategies

Hire a DevOps Engineer, Boost with the DevOps expert

Create a DevOps roadmap with the DevOps Expert before hiring
Get the DevOps expert to interview the candidates
Get advice from the DevOps expert on what to examine during your interview with the candidates
Prioritize the DevOps efforts with the DevOps Expert
Plan the onboarding of the DevOps Engineer with the 6. DevOps expert
Hire the DevOps Engineer
Get advice on managing DevOps projects from the DevOps expert
Discuss the DevOps tasks implementation with the DevOps Engineer and DevOps expert
Occasionally or routinely consult with the DevOps expert on priorities, management, and engineering

Hire a DevOps freelancer, Boost with the DevOps expert

Exactly as the above strategy
Take advantage of the hourly flexibility when needed

Hire a DevOps agency, Boost with the DevOps expert

Create a DevOps roadmap with the DevOps Expert before choosing an agency
Share your requirements with the agency
Filter out any agency that doesn’t provide you with a plan BEFORE work starts
Get the plan reviewed by your DevOps expert: Does it make sense? Is the planned result desired?
Make sure the DevOps agency’s ability to provide support after the work is done
Asses the level of the DevOps agency’s DevOps Engineers with the help of the DevOps expert
Make sure the DevOps agency is flexible with its hourly capacity
Occasionally or routinely consult with the DevOps expert on priorities, management, and engineering

Advantages and disadvantages of the strategies

Option	Good when	Bad when
Full-time + Expert	- Your budget is high - You need flexible DevOps capacity - Your team has DevOps background	- Your budget is low
Freelancer + Expert	- Your budget is low - You need flexible DevOps capacity - You can't risk losing DevOps capacity short-mid-term
Agency + Expert	- Your budget is low - You need flexible DevOps capacity - You want an accurate plan before work starts - You frequently need to support new tools and technologies	- The agency doesn’t have flexible plans - The agency’s team isn’t experienced - The agency doesn’t create clarity before work starts

9th + 10th Consultation Hours: Create a DevOps roadmap + Execute your strategy

You’re ready to create a DevOps roadmap and start executing your strategy!
Creating a DevOps roadmap is the first step of each of the strategies described above.

You had to do a bunch of things first:

You needed a DevOps expert to help you break the vicious DevOps loop - Done.
You needed to set DevOps goals rooted in a deep DevOps understanding and perspective - Done.
You needed to get familiar with different strategies and choose the best one for you - Done!

You’re left with 2 things to do:

1 - Create the DevOps roadmap based on the goals and strategy you’ve set
2 - Start executing the strategy:

Share the DevOps goals with the team
Schedule the interviews with the Engineers/Freelancers/Agencies
Use the DevOps expert to stay on track
Set the milestones for the DevOps goals
Create the tasks
Complete the tasks

Prepare before starting the process

You want to do two things before jumping into the 10 hours:

Before the first 2 hours: Perfect your DevOps terminology (will support rich discussions)
Before the last 2 hours: List existing efforts that might slow down the DevOps roadmap (will help prioritizing)

Ask me anything

If you’re thinking about taking this “10-hours DevOps Pill”, feel free to reach out here.

Do it yourself

You now know the general process for breaking the DevOps loop.
You don’t NEED a DevOps expert.
Feel free to subscribe to my Newsletter and learn how to do it yourself here 👈🏼

The CTO DevOps Handbook: Simple Principles and Examples

Michael Zion — Mon, 25 Dec 2023 21:28:13 +0000

Nail the DevOps part as your company's CTO

The goal of this handbook is to give you clarity on DevOps:

Understand what’s DevOps (in simple words)
Know what’s possible with DevOps (in simple goals)
Get simple “when-to-do-what” DevOps guidelines ‍

I added a bonus at the bottom of the article.
It's a production-ready setup example you could take inspiration from.

Who this article is for

You might be a founder who wishes to get started with DevOps the right way.

You might be a CTO of a 1,000 employees company who wishes to get simple principles.

Or, maybe you’re a Software Engineer, and you want to understand if your company’s DevOps approach is good.

If you’re looking for a simple DevOps playbook, this is it.

Understand the desired result

Two things your company needs to be able to do

Serve its product to customers
Build and improve the product

Abilities you need to build, improve, and serve software

Run experiments and test changes

DevOps has a simple meaning

Developers and Operators have shared responsibility for building and improving the system.

In practice:

Developers are responsible to “Operate”
DevOps Engineers are responsible to enable to “Operate” AND do some of it themselves

Operate = provision, monitor, secure, configure, deploy, scale.

Choose a balance: Enabler, Doer, or Automator

The DevOps role will end up as a balance between:

Enabler: Provides the tools and knowledge to fulfill the DevOps goals
Doer: Does the tasks that fulfill the DevOps goals
Automator: Automates any repeating operation

Know what things you should enable, do, or automate

Provision infrastructure
Secure the system
Deploy workloads
Monitor the system
Recover from issues
Scale up or down
Track & test changes
Automate processes

Choose the right tools

Has state management = Saves time automating state-aware processes (e.g., Terraform)
Has a big community & good docs = Saves time dealing with common issues (e.g., Kubernetes)
Has multiple interface types: API, CLI, UI = Saves time integrating with the existing system (e.g., Vault)

You can also read about choosing tools here.

Set useful goals

There are DevOps goals that adopting them will focus you on the right direction:

One-Click Environments: makes e2e tests easy and quick
Atomic Commits: provides confidence that a tested change will work in production
Separate the Shared & Env-Specific Parts: enables e2e tests as the company scales up

If you want to learn about more useful DevOps goals, feel free to book a free consultation here.

Enablers: Choose the Tools-to-Knowledge Balance

Developers can either have the knowledge or the tools to do something.

More knowledge-reliance: if you want the developers to contribute to the DevOps efforts
More tools-reliance: if you want to abstract the operations from the developers

If the balance between the two is not intentional, it’s accidental.

Doers: Have a good reason to do it

Is it a one-time task?
Does it teach you how the developers work?
Are you directly accountable for the results of the task?

If you answered “no” to the above questions, enable or automate it instead.

Doing more = Learning the system's use-cases

Doing too much = Not scalable, too-much knowledge-reliance

Automators: Have a good reason to automate it

Did it happen before?
Is it likely to happen again?
Will automating it take less time than doing it?
Will automating it teach you an important company process?

If you answered “yes” to 2 out of the 4 questions - automate it!

More automations = Less reliance on knowledge to operate the system.

Too much automations = No system awareness.

P.S. - you can also enable developers to automate it.

Create available DevOps Capacity

The DevOps needs of a company have spikes.

One month you need 2 DevOps Engineers, and half of that the next month.

Switchovers between big efforts and small tasks are common.

This is true, especially for new companies.

Break the assumption: “DevOps tasks must be done by a DevOps Engineer”.

There are 3 types of DevOps capacity

Non-Flexible: A full-time DevOps Engineer on the team
Semi-Flexible: Key developers that can contribute to the DevOps goals
Fully-Flexible: A flexible DevOps Services company or freelancer

You can read more about calculating the DevOps capacity your company needs here.

When to focus on what: Common Dilemmas

When: You work alone, and the system is simple

‍Focus: On simplifying the development - Dockerize your apps, Create a post-commit pipeline that runs tests

When: You need to be able to create new environments quickly (for development, or for clients)

‍ Focus: On implementing “One-Click Environments”: Using IaC (e.g., Terraform) + Deployment tool (Depends on the platform).

When: You want to e2e test every code modification, but there are many code modifications

‍> Focus: On splitting the “One-Click Env” into a “base” with shared resources, and “env” with env-specific resources

When: You want to unify & standardize how you deploy, monitor, scale, configure, and secure your workloads

Focus: On implementing an orchestrator such as Kubernetes

When: You want you have many moving parts and wish to be certain a tested change will work

‍ Focus: On implementing GitOps and consider a Monorepo (the sooner the better)

When: You want the DevOps efforts to be done by the dev team

‍ Focus: On using “actual” IaC tools (Pulumi Typescript/Python), Full “how to operate” (see above) documentation‍

Never: Invest lots of time in new tech without a strong reason

Always:

Have your code in Git
Monitor the basic stuff: CPU, Memory, Disk, Network, App Logs, Cloud Costs
Architect for high-availability
Test before you deploy

BONUS: An example setup for a CTO approaching Production

2 AWS Accounts

One for development and staging
Another for production

Monorepo in Github

Docker-Compose for local development

2 Infrastructure-as-Code projects: 'base' & 'apps'

base = shared resources (e.g., VPC, RDS, ECS Cluster, EKS Cluster)
apps = env-specific resources (e.g., Lambda Functions, ECS Services, Kubernetes Namespaces)
config file per environment

Github Actions Workflow: Development workflow

Checkout branch and locally develop + test changes
Create a Pull Request: Deploys a Pull-Request ‘apps’ environment on the ‘development’ environment ‘base’
On merge to main: Deploys from the ‘main’ branch an ‘apps’ environment onto the ‘development’ environment ‘base’
Manual: Deploy from the ‘main’ branch onto the ‘staging’ / ‘production’ environment ‘base’

Setup Notes

Avoid mentioning an environmnent's name in the code for conditional resources deployment
Use each environment’s config file to declare if a resource should be created
Could be implemented using Terraform, Terragrunt, Pulumi, CDK, and other IaC tools
Production should have 2-instances of every workload for high-availability

If you’d like to see this setup in your startup, click here to book a call 👈🏼

P.S. - I'll be updating this page occasionally, so you might want to visit again

Another Bonus: DevOps Dictionary for Human Beings

Term	Definition	Tools
Environment	A working instance of the entire system
CI (Continuous Integration)	Enable developers to collaborate by agreeing on a single source-of-truth (master/main)	Jenkins, Github Actions, GitlabCI
CD (Continuous Delivery)	Create an artifact that’s ready for production (tested, tagged)	JFrog Artifactory, Nexus, AWS ECR
CD (Continuous Deployment)	Every available deliverable (artifact) gets deployed automatically	ArgoCD, Jenkins, AWS CodeDeploy
Monitoring / Observability	Collect metrics/traces/logs from apps and infrastructure, analyze them, and display them, and setup alerts	Prometheus, Jaeger, Elasticsearch, Fluentd, OpenTelemetry
Infrastructure	The resources on which the workloads run, in which the data is stored, and through which the network flows	Servers, Databases, Network Routers & Switches
Cloud Infrastructure	Same as the above, but specifically in the cloud	AWS EC2, AWS RDS, GCP Compute Engine, Azure Virtual Machines
Cloud	Computing & Data services served from remote locations for you to build your system	AWS, Azure, GCP
Containerization & Virtualization	Technologies utilizing Kernel & OS features to create virtual machines, or isolate process (AKA run containers)	Docker, vSphere, KVM
Secrets Management	Storing and retrieving sensitive configurations (e.g., tokens, passwords)	Hashicorp Vault, AWS Secrets Manager, SealedSecrets
Configuration Management	Usually refers to preparing servers for workloads (e.g., creating directories & files, starting processes)	Ansible, Chef, Puppet
Version Control	Saving the code in a versioned way (Git)	Github, Gitlab
GitOps	Making the system is the same as it’s described in Git	Flux, ArgoCD, Jenkins
Monorepo	All of the company’s code is in one Git Repository	NX, Turborepo
Polyrepo	Multiple Git repositories for different components
IaC (Infrastructure-as-Code)	Creating Cloud infrastructure with idempotent code and state management	Terraform, Pulumi, CDK, Crossplane
Deployment	Execute, serve, or install the artifacts	ArgoCD, Jenkins, AWS CodeDeploy, Scripts (Bash, Python, etc.)
Orchestrator	Dynamically allocating workloads to a pool of nodes	Kubernetes, Nomad, AWS ECS
Authentication & Authorization	Making sure each person, workload, or resource, has access only to what’s necessary (other workloads and resources)	AWS IAM, OpenID, OpenVPN, Twingate, Istio
Service Discovery	Exposing available workloads using DNS	Consul, CoreDNS

Get more practical advice

I post small nuggets of practical advice on the "MeteorOps Newsletter".
You can subscribe here 👈🏼

One-Click Environment: The Ultimate DevOps Goal

Michael Zion — Mon, 18 Dec 2023 15:13:05 +0000

I remember when I started working as a DevOps Engineer.

For me, ”Best-Practices” was the most sacred phrase.

I searched for best-practices not only when I wanted to know how to do something. I searched for best-practices to get inspiration for what I should do.

In hindsight, big mistake. It happened because I didn’t have a clear goal.

When you don’t have a clear goal, you’ll try to see what the experts do and copy. You’ll end up embracing their best-practices without knowing why they did them.

I’ll share with you here why “One-Click Environment” is the ultimate DevOps goal.

As a bonus, I’ll share a framework to achieve it fast.
‍

The ultimate DevOps goal - One-Click Environment

Looking back, the goal you should set is simple: Create an environment in “One-Click”. I’m saving you years of confusion that I had to go through.

Why is this the ultimate goal, you ask? Three reasons:

It makes it easy to test changes
It makes it clear what should be automated
It improves the system’s recoverability

What do I mean by “Environment”?

A ‘clean’ and working version of the entire system.

The most popular environment is the Production environment, which serves the clients.

There are static testing environments, such as ‘dev’, ‘staging’, ‘uat’, and more.

There are also ephemeral testing environments, such as Pull-Request-Environments, Git-Branch-Environments, Developer-Environment, etc.

Why it makes it easy to test changes?

The company with the fastest development speed I ever saw was a supply-chain startup. They deployed high-quality code to production a lot, every day, with no downtime.

How? We deployed ephemeral environments for every pull-request. The environments were identical to production. After running automated tests they terminated.

We used Pulumi (Typescript) on AWS and documented how to change the environment. This way developers took part in contributing to the environments.

Programming is not Mathematics

Edsger Dijkstra, was one of the pioneers of computer science. He envisioned programming as a mathematical discipline. He thought programmers will mathematically prove their code works.

When was the last time you wrote a mathematical proof for your code? Probably never.

Programming is Science

Instead, you took the scientific approach: You ran some tests to see if your code works as expected. And how do you run tests? On an environment of course!

The easier it is to create an environment, the faster you’ll be able to run your tests. The ‘cleaner’ your environment is, the more trust you’ll have in the test results.

Why it makes it clear what should be automated?

Another startup we work with has 15 Kubernetes clusters with 2,200 nodes on GCP - All created manually!

Developers asked for support many times a day; "Create this DB", "I need a new NodePool", etc.

With every new request, we had to ask - "Should this be automated?"

We ended up automating everything required for every new environment.

How? We created a Pulumi Typescript codebase and imported the entire infrastructure into it. On top of that, developers felt comfortable contributing to it, as it was a Typescript project.

The “should this be automated?” mental struggle

I asked myself at least 1,836 times “should this be automated?”, on lots of things. Sometimes I decided to automate nonsense.

If you want to understand if it's worth automating, ask yourself 2 questions:

Is it going to happen again?
Will automating it take less time than doing it manually?

Don’t rely on your memory

You’ll need to remember every ad-hoc modification you did to an environment. If a change isn’t done as part of the “One-Click Environment” automation, you’ll have to rely on your memory and do it again.

Imagine a developer creating a new environment, running its tests, only to find out the DB isn't there. Someone created it manually in the other environments.

Bottom line, how do “One-Click Environments” guide automation?

Next time you ask yourself “should this be automated?”

If it’s part of what a new and clean environment needs: Easy YES
If not: A strong maybe

By the way, maybe you want to automate it, but you’re drowning in so many requests that it’s impossible.

In that case, send your management this article.

It's about calculating how much DevOps capacity your company needs.

Maybe it’ll help understand how much DevOps capacity you need.

Why it improves the system’s recoverability?

An IoT startup we worked with had 5 clients, with an AWS account per client.

It was supposed to be the same system per client, but it wasn’t!

Each account’s production was created with Terraform (Terragrunt) and Ansible. But, there were so many ad-hoc changes, it took about 2-4 days to create a full environment from scratch.

We decided to use Jenkins and created a pipeline that automated everything. When we finished, you could deploy a full environment from scratch in 50-minutes. (after some parallelization it went down to 20-minutes)

You should have heard the sigh of relief the team had when the demo finished — ”No more fear of not recovering from a production incident!”.

Create a new environment and route traffic to it

Think about your production — what would it take to create it from scratch? If your answer is “I don’t know”, it’s most likely because some things were created manually or ad-hoc.

If your answer is “one-click”, we are hiring. Please click this link asap: Senior DevOps Engineer Position at MeteorOps.

Imagine a production failure that requires creating a new environment, and finding out you can't. Some things were modified manually.

Or, as the poet Capone-N-Noreaga wrote - “Oh no, oh no, oh no no no no no”.

When you have “One-Click Environments”, you can deploy a new environment fast and use it instead of the old one.
‍

It also improves high-availability

When you can create an environment from scratch, things become easier:

creating a multi-region setup
creating an active-active setup
creating an active-passive setup ‍ You can deploy many environments in different locations, and route traffic to an environment based on your needs.

TPCS - A useful framework to achieve a One-Click Environment

Tools, Pipeline, Contributions, State

‍

It all boils down to 4 things you should be doing:

You should choose the tools to: manage data, provision, deploy, configure, secure, test, monitor, and automate
Then you should build the pipeline that uses the tools to create the environment
You should also make it easy to contribute to the pipeline that creates an environment
And, you should continuously reconcile the state of the system to its desired state

Principles for choosing tools

Choose tools that answer the below requirements:

More native integrations per tool, less glue required — Makes building the pipeline easier (e.g. - AWS)
Popular tools with active communities make documentation easier (e.g. - Kubernetes) Tools with a declarative syntax are self-documenting and save you some hustle (e.g. - Terraform)
Tools with native state management will save you time reconciling the state of the system (e.g. - Terraform)
Tools with extensive APIs will make it easier to give the developers more ownership (e.g. - Pulumi)

Guidelines to build the pipeline

This is where you’re flexibility is your number one priority:

Choose a flexible automation platform (e.g. - Jenkins)
Create one pipeline and make it possible to pass parameters to it
Manage the pipeline’s code straight in Git
Bonus: Parallelize its stages to speed it up ‍ ### Make it easy to contribute
Describe the desired state of the system in a Git repository, and make it easy for developers to contribute
Make it clear where to find infra-related stuff, deployment-related stuff, etc
Write good documentation ‍

Good documentation answers these questions:

How to provision a resource?
How to deploy a workload?
How to configure the system?
How to secure the system?
How to test changes?
How to monitor a workload?
How to automate a process?
How to manage the data’s lifecycle? (e.g. - Run DB migrations) ‍ ### Reconcile the state of the system This step is made easier if you chose idempotent tools with state management, and made harder if you didn’t! (choose idempotent tools)

The guiding principle here is simple. If you run the pipeline twice with the same parameters, nothing should happen the second time.

That’s it, you’re good to go!

But first…

‍

A Recap!

Without a useful goal, you might embrace “best-practices” that aren’t good for you.
The most useful goal for a DevOps Engineer is being able to create an environment with “One-Click”.
Because it makes it easy to test changes,
And It makes it clear what should be automated,
And It makes it easy to recover the system.
A framework that helps achieve this goal is TPCS: Tools, Pipelines, Contributions, State ‍

Whenever I think about going forward without a goal, I’m reminded of this gem from ‘Alice in Wonderland’:

“Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go.
Alice: ...So long as I get somewhere.
The Cheshire Cat: Oh, you're sure to do that, if only you walk long enough.”
― Lewis Carroll, Alice in Wonderland
‍

If you want to implement the “One-Click Environment” in your company
You should know we (MeteorOps) took many companies from “no-automation” to “one-click environment”. If you’re interested in a free consultation about it, feel free to click here 👈