DEV Community: Christian Melendez

Working with Multiple Kubernetes Clusters using Okteto

Christian Melendez — Wed, 16 Dec 2020 22:25:21 +0000

As a developer, there's going to be a time where you need to work with different Kubernetes clusters. At a minimum, you need to work with two clusters: one is your local cluster, and a second one could be a remote cluster for testing purposes in a cloud provider or elsewhere. Kubernetes has contexts, a set of configurations you can use to interact with different clusters from your workstation using kubectl. At the end of the day, remember that you're interacting with an API to persist the desired state for your workloads.

Even though kubectl offers a set of commands you could use to work with different clusters, you might need another set of tools to make your life easier. Sometimes, typing long commands would decrease your productivity, and they're hard to remember initially. And if you're using okteto, you don't need to use another tool and almost forget about kubectl to work with different clusters. We'll see in this post how's that possible.

But first, let's briefly talk about contexts in Kubernetes, which is the configuration you use to work with multiple clusters.

What's a Kubernetes Context?

A context in Kubernetes a set of parameters you have to communicate with a cluster from your local workstation using kubectl. For instance, you could have something like this in your ~/.kube/config file:

contexts:
- context:
    cluster: docker-desktop
    namespace: default
    user: docker-desktop
  name: docker-desktop

Each context contains a cluster, namespace, and user information that kubectl uses to make the cluster's API calls. You can find all these configurations in the ~/.kube/config file. However, configuring this file could be overwhelming if you're starting with Kubernetes, especially if you want to start coding your applications. Most of the time, you don't have to configure the kubeconfig file as most of the Kubernetes installation providers already offer a command to generate that file.

If you're already using Kubernetes, you might give it a look in your ~/.kube/config file, and you might find out that you already have the configuration necessary to connect to different clusters. In case you want to understand more about Kubernetes contexts, you can visit the official Kubernetes documentation site and learn how to configure access to multiple clusters using contexts.

In this post, I'll focus on the different ways you have to use contexts to increase your development productivity when working with more than one cluster. Also, I'll be using Docker for Desktop because it comes with a local Kubernetes cluster that you can enable with only a few clicks.

Using Kubernetes Contexts

Once you have your kubeconfig file configured with the contexts you'll need, it's time to use them.

The idea behind configuring a Kubernetes context is that once you have configured your access to the different clusters, you can run this command:

$ kubectl config use-context docker-desktop
Switched to context "docker-desktop."
$ kubectl config use-context remote-cluster
Switched to context "remote-cluster".

However, the above commands are still long and hard to remember. Although you could create aliases in your workstation, I'd highly recommend you to use kubectx instead. Kubectx is a command line tool for switching and managing Kubernetes contexts. Besides changing contexts, you can easily go back to the previous contexts, show the current context, rename or delete a context, and more. So the above commands now will be:

$ kubectx docker-desktop
Switched to context "docker-desktop".
$ kubectx remote-cluster
Switched to context "remote-cluster".
$ kubectx -
Switched to context "docker-desktop".

Every subsequent command you run will be making calls to the corresponding Kubernetes cluster.

Working with Multiple Namespaces

At this point, you're able to connect and work with multiple Kubernetes clusters from your workstation. But remember that a context is also referring to a namespace in Kubernetes. That namespace will be default when you don't specify a namespace in a kubectl command. For instance, your docker-desktop context has the default namespace configured. Then, every time you run kubectl get pods, it will return the list of pods from the default namespace.

However, there are going to be times where you also need to work with different namespaces. Perhaps you're in charge of two applications, and each application uses a namespace to deploy the Kubernetes objects it needs. Every time you run a kubectl command, you'd have to specify the name of the namespace you want to use, like this:

$ kubectl get pods -n service-a
$ kubectl get svc -n service-b

You might not see a problem with the above commands. However, when you're coding, you might need to change namespaces a little bit quicker and avoid any confusion if you forgot to include the namespace's proper name. If you don't want to specify a namespace in every kubectl command you run, you'd have to modify the ~/.kube/config file. However, there's a project called kubens (from the same project creator as kubectx) that you could use to switch between different namespaces quickly. kubens will update the Kubernetes context for you automatically. The above commands now can be like this:

$ kubens
default
service-a
service-b
$ kuebns service-a
Context "docker-desktop" modified.
Active namespace is "service-a".
$ kubectl get pods
$ kubens service-b
Context "docker-desktop" modified.
Active namespace is "service-b".
$ kubectl get svc

As you've noticed, kubectx and kubens help you work with different clusters and namespaces, which is a little less problematic. But still need to type a lot of commands.

But what if there's even a better way for developers? Let's see.

Developing with Contexts in Okteto

Okteto has support for configuring the context and namespace your applications might need. Hence, you have consistency at coding time because all developers are going to use the same configurations. Also, you'll have fewer commands to run, so you can focus on writing your application, not on managing clusters. Even if the existing commands from kubectcl or the kubectx utility, you still need to switch manually. Conversely, with Okteto, you can include this configuration in the YAML manifest and forget about it.

For instance, following our sample from Rust, you could modify the okteto.yaml to this:

name: hello-world
image: okteto/rust:1
command: ["bash"]
workdir: /okteto
context: docker-desktop
securityContext:
  capabilities:
    add:
    - SYS_PTRACE
forward:
  - 3000:3000
  - 2345:2345
persistentVolume:
  enabled: false

Notice how we're specifying which context to use with the line context: docker-desktop. And even if you're currently pointing to the remote cluster context, Okteto will use the Kubernetes cluster from the Docker Desktop app. Additionally, if you don't want to have hardcoded values, you could use an environment variable, like context: $APP_CONTEXT. Developers can have different names for a local context, and they only need to set up $APP_CONTEXT environment variable so that Okteto points to the proper cluster.

Let's give it a try, run the following command to force the usage of the remote cluster:

$ kubectx remote-cluster
Switched to context "remote-cluster".

Now that the okteto.yml has the context configuration, run the okteto up command and compile the application, like this:

$ okteto up
 ✓  Development container activated
 ✓  Connected to your development container
 ✓  Files synchronized
    Context:   docker-desktop
    Namespace: default
    Name:      hello-world
    Forward:   2345 -> 2345
               3000 -> 3000

Welcome to your development container. Happy coding!
default:hello-world okteto> cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.78s
     Running `target/debug/hello-world`
Listening on http://0.0.0.0:3000

Notice how the Okteto output tells you which context is using.

You should now be able to see the application running in the local cluster, like this:

$ curl localhost:3000
Hello Kubernetes!

When you close the Okteto terminal, notice that you're still using the remote cluster context. Locally, nothing changed because Okteto automatically knows which context to use to deploy your application where it should be running. With this, you can rest assured that you're developing in a local cluster and not in a production cluster!

Configuring Namespaces in Okteto

Additionally, Okteto allows you to change the namespace you'd like to use and follow the same principle: giving fewer things to worry about to a developer. Like the way you configure the context property, you also have the namespace property. So, your okteto.yml manifest could look like this:

name: hello-world
image: okteto/rust:1
command: ["bash"]
workdir: /okteto
context: docker-desktop
namespace: hello-rust
securityContext:
  capabilities:
    add:
    - SYS_PTRACE
forward:
  - 3000:3000
  - 2345:2345
persistentVolume:
  enabled: false

To give it a try to this configuration, let's go back to using the local context and make sure to set up the default namespace, like this:

$ kubectx docker-desktop
Switched to context "docker-desktop".
$ kubens default
Context "docker-desktop" modified.
Active namespace is "default".

Also, create a new namespace called hello-rust where you'll deploy your application:

$ kubectl create namespace hello-rust
$ kubectl apply -f k8s.yaml -n hello-rust

When you run okteto up, and then cargo run, your application will be running in the hello-rust namespace, even if your workstation uses the default namespace. You could confirm that nothing is running the default namespace by running the kubectl get pods outside of the Okteto terminal.

$ okteto up
 ✓  Development container activated
 ✓  Connected to your development container
 ✓  Files synchronized
    Context:   docker-desktop
    Namespace: hello-rust
    Name:      hello-world
    Forward:   2345 -> 2345
               3000 -> 3000

Welcome to your development container. Happy coding!
hello-rust:hello-world okteto>

Notice from the output that Okteto tells you in which context and namespace it's running.

Conclusion

Kubernetes already has the tooling you need to work with multiple clusters and namespaces. However, this could affect a developer's productivity and is still error-prone. You might be pointing to the wrong cluster or namespace, and you didn't know. When you use Okteto for developing your applications, you can make sure the applications will use the proper context and namespace even if your kubectl has different settings.

Getting Started with Okteto and Rust

Christian Melendez — Tue, 01 Dec 2020 14:53:20 +0000

Without any doubt, Kubernetes has become the default platform to run modern applications. However, for a developer, working with Kubernetes brings a new set of challenges, and a learning curve that might be intimidating. Additionally, and more importantly, the development inner loop now requires a few more steps before you can test your app.

To put this in perspective, prior to Kubernetes, the inner loop for Rustaceans used to look something like this:

Write some code
Run the app with cargo run (which includes compiling)

And when you add Kubernetes to the formula, the inner loop looks like this:

Write some code
Build a container image (which includes compiling)
Push the container image to the registry
Deploy the app to Kubernetes

These extra steps from above easily translate into a minimum of two minutes. If you want to fix something quickly, those extra steps become an eternity. There has to be a better way, right? Yes, there is, and the answer is Okteto.

Okteto: A Rustacean’s Friend

Okteto is a tool that helps developers to develop applications for Kubernetes without any extra steps in the inner loop. Rustaceans immediately forget that they’re not running applications locally but in a Kubernetes cluster.

Long gone are the days where you defer for later the stage of testing your applications in Kubernetes. Okteto helps you continue developing with your existing inner loop and testing the app in Kubernetes without any extra steps. Once you have Okteto installed locally, when you run cargo run, Okteto takes care of the rest.

But, I know you came here for the details, so let’s see how you can get started with Okteto and Rust.

Step 1: Deploy a Simple Rust Application to Kubernetes

You can use any existing Rust application, but if you don’t have one, clone our sample app repository also to get the manifest to deploy the application in Kubernetes:

$ git clone https://github.com/okteto/rust-getting-started  
$ cd rust-getting-started

Before you start running your application locally, let’s confirm that you can deploy it to Kubernetes. The repository comes with a manifest you can use to deploy the app to Kubernetes. Any Kubernetes cluster works; it doesn’t matter if you’re using a local cluster or a remote one running a cloud provider. However, if you don’t have a Kubernetes cluster, you can give Okteto Cloud a try

Deploy your application to Kubernetes running the following command

$ kubectl apply -f k8s.yml

You can confirm that the application is running by getting the pod’s status, which for now is enough.

Step 2: Install the Okteto CLI Locally

To get started, you need to install the Okteto CLI on your local workstation, where you type your Rust code.

The Okteto CLI is an open-source project that helps you to continue using your local tooling. Once you finish writing code, you can run “cargo run” with Okteto, and the app container is updated in Kubernetes. Okteto is the tool that helps you to improve your developer productivity when developing apps that run in Kubernetes. So, it would help if you have the Okteto CLI locally.

If you’re using Mac or Linux, run the following command to install the Okteto CLI:

$ curl https://get.okteto.com -sSfL | sh

If you’re using Windows, you need to download the executable and include its location in your $PATH variable. Or, you can also use the Linux subsystem and run the command from above.

To confirm that the Okteto CLI works on your machine, run the following command:

$ okteto version

Great, you’re getting closer to experience getting back your traditional inner development cycle.

Step 3: Create an Okteto Manifest

Before you can start using Okteto, you need to create a YAML manifest required to activate your development environment.

You don’t have to learn how to create an Okteto manifest yet. Instead, use the Okteto CLI, which will ask you to pick an existing deployment in your Kubernetes cluster (this is why you started by deploying the application). Based on the code you have, the CLI generates a manifest. So, run the following command:

$ okteto init

You should see an output like this one:

This command walks you through creating an okteto manifest.
It only covers the most common items, and tries to guess sensible defaults.
See https://okteto.com/docs/reference/manifest for the official documentation about the okteto manifest.
Use the arrow keys to navigate: ↓ ↑ → ←
Select the deployment you want to develop:
  ▸ hello-world
    Use default values

The hello-world deployment is the one you deployed previously for the sample Rust app.

And now you should have an okteto.yml file with the following content:

name: hello-world
image: okteto/rust:1
command: ["bash"]
workdir: /okteto
securityContext:
  capabilities:
    add:
    - SYS_PTRACE
forward:
  - 3000:3000
  - 2345:2345
persistentVolume:
  enabled: false

Without explaining every line of the above manifest, let’s highlight and explain a few of them:

name: the name of the Kubernetes deployment you want to put on “development mode.”
image: the image used by the development container
command: the start command of the development container
forward: a list of ports to forward from your development container.

You can get more details about the Okteto manifest at our official documentation site.

Step 4: Activate Your Development Container

Now you have everything you need to spin up your development environment using Okteto.

Run the following command to enter in a development mode:

$ okteto up

You should see something like this:

✓  Development container activated
✓  Files synchronized
   Namespace: default
   Name:      hello-world
   Forward:   8080 -> 8080
              2345 -> 2345
Welcome to your development container. Happy coding!
default:hello-world app>

This becomes your new terminal to run your Rust application every time you change its code. All the commands you run in this terminal are running in Kubernetes. Okteto opens a terminal in a container that has all the tooling you need for developing in Rust. To give it a try, run the following command:

default:hello-world okteto> cargo run

You should see an output like the following where you can see how the app is compiling and launching a proxy:

Compiling hello-world v0.1.0 (/okteto)
  Finished dev [unoptimized + debuginfo] target(s) in 4.72s
     Running `target/debug/hello-world`
Listening on http://0.0.0.0:3000

To confirm that the app works, launch a new terminal, and run the following command:

$ curl localhost:3000  
Hello Kubernetes!

You might be thinking that you’ve just complicated things a little bit, but hold on a second and keep reading.

Step 5: See Application Changes in Kubernetes

Don’t close the Okteto terminal, and let’s change something in the Rust sample app.

For instance, head over to the src/main.rs file, and in line #9 change the response text, like this:

async fn hello(\_: Request) -> Result&lt;Response, Infallible> {  
 Ok(Response::new(Body::from("Hello World from a Kubernetes Cluster!")))  
}

Go back to the Okteto terminal, cancel the command (Ctrl + C) and rerun the cargo command (as you would typically do if you were testing your application locally):

default:hello-world okteto> cargo run

Go back to the other terminal where you’re testing your application, and rerun the curl command:

$ curl localhost:3000  
Hello World from a Kubernetes Cluster!

See? You didn’t need any additional steps to see the changes reflected in Kubernetes.

Conclusion

And that’s it. This is how you can recover your developer productivity when you need to test your apps in Kubernetes. You don’t need to extend your inner development loop because Okteto helps you to test remotely in the cluster. Rather than making you recreate your cluster on your laptop, Okteto transparently moves your inner loop to the cluster, while still letting you develop locally.

Visit our website to learn more about how to improve your team developer productivity with Okteto. Follow us on Twitter and join our #okteto channel in the Kubernetes community Slack to share your feedback with our community.

Happy koding!

5 Critical DevOps Practices

Christian Melendez — Tue, 07 Jan 2020 16:41:14 +0000

DevOps is like pizza. We can't think of pizza without considering critical ingredients: dough, sauce, cheese, and your preferred choice for vegetables and proteins. Everyone likes different toppings. In my case, I can't think about pizza without extra cheese and meat. You might choose differently, but I think we can agree there are some ingredients that are critical for this food to be called pizza. Quality and ingredients will vary, but some things will always remain true.

Well, it's the same with DevOps practices. There are some critical practices, and you can't think about DevOps without considering them. Everyone will have preferred choices regarding the tools and the process, but the practice will remain and each practice complements the other.

Every critical DevOps practice takes time to get down, but the end result will be magnificent. So, let's discuss what they are and how to implement them.

1. Involvement From Everyone In the Process

Group involvement is like the dough of our pizza. It's the foundation, the base. This is the DevOps practice that you need to start before any other. Gone are the days when you would define DevOps as a collaboration between just development and operations. What about QA? What about SecOps? And more importantly, what about management? In order to get a good base for DevOps, you have to stop thinking about teams or people. Everyone involved in the process of developing the product should have a mindset of collaboration.

If this doesn't happen, you'll constantly have people questioning everything or ignoring important aspects of the project. It's not like you're working with saboteurs—no one's trying to destroy the initiative. But people have to understand the impact of their actions. When you involve everyone in the process, people can grasp the whole picture. And just like that, you've laid the foundation for DevOps.

2. Trust Only What You Have In Version Control

Trusting only what you have in version control is like the sauce of our pizza. It's the first ingredient you put on it. The flavor will vary, but it's a critical practice.

At work, every time someone interrupts me with a question about a certain functionality—especially one that I coded a while back—I answer with what I remember, but I always add, "Don't trust me, though. Let's go and see what's in the repository." Now it's not just code. You'll have to put in version control infrastructure specifications, dependencies, and configuration templates. Some of those configuration values will be visible to everyone, and others won't. It becomes critical to have it just for the sake of repetition. Repetition and consistency are key.

The only source of truth should be what you have in version control. Avoid doing manual changes. If something needs to change, it should be done here. Not anywhere else. The team will then know the "what," "when," "why," and "who" when it comes to changes, and that knowledge is the difference between a team with poor communication and a high-functioning DevOps team. As this practice becomes routine, team confidence will increase. There's nothing like trusting the process.

3. Be Confident With Automated Testing

Cheese! Automated testing is like cheese. And there's no pizza without cheese. You can't afford to not have the practice of automated testing in place.

Forget about DevOps for a moment. This practice is critical now more than ever in our industry. If you want to succeed and deploy with confidence, a good suite of tests you can trust is crucial. You need tests that will fail if the app is not providing the expected output. Otherwise, the team will be slower to call something is done, thanks to manual testing.

There are several types of tests that you can choose, and today's not the day to talk about them. But know that the speed of the tests is a critical factor. If it takes too much time to receive feedback, everyone will avoid them. Some people might go to the extreme of commenting them out or removing them. As long as almost no human intervention is needed and they run fast, your tests are good.

Without the practice of automated testing, all DevOps efforts will be compromised because you'll instead be dealing with a constant barrage of errors.

4. Regularly Integrate Your Codebase

The regular integration of your codebase is like our pizza's vegetables. In order to have a balanced diet, you need to include them. As kids, we didn't like them that much, but as adults, we realized that we need them. You might not like this practice at the beginning, but you will after time. This practice consists of regularly integrating your codebase into the mainline, trunk or master (depending on the version control tool). Avoid long-lived feature branches, and integrate your code at minimum once a day. Everyone should always have the same working copy of the codebase each morning.

Things can go wrong, I know. That's why, in addition to integrating your codebase, you'll need to have an automated way to check if everything went well. Compile the code, do your static code analysis, check that all unit tests passed, and then pack to deploy the same code everywhere. You'll build only once. If you don't do this, you'll add more time to delivery, and you might encounter new problems every time. This practice might sound crazy if it's the first time you've heard of it.

Architecture plays a big role here. You will need to have a loosely coupled one. Things need to be easy to change and evolve all the time.

Are you still in doubt that this is necessary? I understand. The first time I read about this practice, I thought, "It's not for me." Try it out and see the difference it can make. You can leverage feature flags to always have the option of turning off what you've checked in.

5. Deterministic Deployments in All Environments

Deterministic deployments in all environments are like proteins. A good pizza will have them, and so will a good DevOps implementation. I never get tired of telling people that deployments should become a boring task: predictable, without randomness. You'll need to have the above practices in place in order to practice this one, though. It's about having a deployment pipeline. There are gates that require the human skill of critical thinking in order to continue.

It's of the utmost importance that the delivery pipeline is the same in every environment. Remember, you don't want to be surprised on release day. Development, QA, staging, and production should have the same pipeline. For development, you'll want everything automated without anyone approving. In QA, you'll want to have a bigger suite of tests approving it. Staging might be the same but with a different suite of tests, usually that take more time to validate. And for production, it should be a business decision.

Let me tell you also that it's not just about code. It's about everything involved in the process of deploying the app in a working state. This means that aspects like infrastructure and configuration should be considered too. Processes like infrastructure as code and configuration management are critical. Of course, automation will always be key because the idea is to provide value to your customers in a timely manner. That's why this practice is critical. You can't let fate take care of this. There's nothing worse than calling things done when developers have delivered.

It's All About Providing Value

As I've been saying, it's all about pizza. Wait, sorry. It's all about providing value with this set of practices.

It's important to have at least these five elements in place in order to succeed. Practicing these will help to increase your team's confidence more and more with each release. Persevere through the pain of implementing something new, because the more you practice these, the better you'll become. Customers will be impressed when you're mature enough in DevOps that you deliver value to them—and in a timely fashion—each time they make a request.

Sexy But Useless DevOps Trends

Christian Melendez — Tue, 17 Dec 2019 16:43:50 +0000

What's sexy but useless? A Ferrari in a traffic jam. It's beautiful, but all that power means nothing. When trapped in traffic, it can't live up to its full potential.

Same with DevOps. While there are some critical DevOps functions that you absolutely need, there are some sexy but useless DevOps trends that are good to be aware of. Truth be told, there's no recipe that will tell you how to succeed in DevOps. Everyone will have different opinions, and what worked for others might not work for you. But you can trust one thing: there are some actions that will guide you directly to frustration with DevOps.

With the amount of information out there about DevOps, you might get overwhelmed and think it's not for you. You also might think the learning curve is too steep---that you need to change too many things before you get started. Maybe you'll need a new team, new tools, more metrics, more time... you name it.

My advice is this: don't get distracted by all things that people say about DevOps. These things I'm going to talk about here, for instance, are all style and no substance.

1. Having a Separate DevOps Team

The idea of DevOps is to remove silos. It doesn't make sense then to include another team that will be in charge of handling or easing the communication between developers and operations. Historically, these two teams have had conflicts, so adding a DevOps team is similar to adding therapists for each team. Having a DevOps team might look sexy because now you can tell the world you're doing DevOps. Your team will hurry to update their LinkedIn profile. But in reality, you're just adding more complexity.

What would a potential problem with a DevOps team look like? Well, developers and operations could start saying, "It's not my job to automate the release; it's the DevOps team's responsibility." And what about accountability and ownership? You'll lose it.

Another downside is that you'll need to have a "full-stack" team that has a background in development, operations, testing, and security. I don't doubt there are people with these skills, but you lose a lot when someone isn't specializing in one thing.

So instead of having a separate team that will create more handoffs, what you can do is identify where the problem really is. It could be as simple as changing a culture that doesn't share knowledge. Are developers thinking about using a new NoSQL database? Great. The developers should let operations know so they can learn it too. Is operations thinking about adopting containers? Nice. They should let developers know about it---I bet the devs would appreciate a production-like environment.

So, please, don't add a new team to your DevOps initiative. Say no to silos. Instead, target the real problems: the ones that made you think you needed a DevOps team in the first place.

2. "Going DevOps" By Virtue of Adopting Some Tool

As the great Grady Booch once said, "A fool with a tool is still a fool." While using the fanciest, most-almighty-seeming tool for DevOps might seem sexy, it's useless if you're hoping it will magically make you a practitioner of DevOps. Same for things like cloud, microservices, or containers. You don't need any of those things to provide value. If you don't know what problem a tool or technology stack is solving, ask yourself why they even exist.

This doesn't mean that tools aren't important and they can't be useful. Some of them will help you speed up, but you shouldn't fully depend on them. They're always evolving.

In fact, you can embrace DevOps without tools, and James Shore's post on how to do continuous integration with the bare minimum sets a baseline for what I'd say about tools and DevOps. The post basically says that you only need an old computer that you're not using anymore, a rubber chicken, a desk bell, a script, and Kool-Aid to do continuous integration. Sounds silly, but it helps you understand that you don't need a tool to start things; you can use what you already have in place. You just need to be creative.

3. Having Developers Manage Production Environments

You might be tempted to do this, especially after hearing that companies like Netflix and Amazon are letting their developers push changes directly to a production environment several times a day. But you shouldn't forget that, in order to achieve this, the delivery pipeline needs to be deterministic and accompanied by a trustworthy set of tests. While I believe that DevOps encourages freedom, I don't think giving the keys to production to everyone will be beneficial. Folks have been fired on their first day of work because they were allowed---and proceeded---to screw things up badly. Imagine giving everyone at a bank a key to the vault. It's like that.

I'm not saying that developers shouldn't be trusted, nor am I saying that companies that give full permissions to developers are nuts. It's just not for everyone. Giving this type of access could make you ask why you need an operations team.

Instead, consider giving developers indirect access to production. By indirect access, I'm saying the operations team should provide all the tools developers need to make better decisions. For example, you could give developers read-only access to the system you're using to administer infrastructure (e.g., AWS or vSphere).

You could also give developers access to the log management tool or to whatever you use to store infrastructure metrics. If developers have visibility into things like CPU, memory, requests per second, error rates, and other such things, it will help them be accountable. You don't need to give them full permissions. Just give them the proper tools to make their work easier. And in the end, remember that operations folks have way more to do than just releasing and operating systems, and most people are just trying to help.

4. Giving Yourself X Months to Become "DevOps Compliant"

Are you looking for a crystal ball that can tell you when you'll finish with your DevOps initiative? That would be Ferrari-style sexy. You could, in fact, reach the point of doing several deployments per day, being always ready to deliver. That sounds next-level sexy, especially to top-level management. But the ambition to finally have "arrived," as far as DevOps is concerned, is fool's gold. This should be something that you constantly improve upon. What if suddenly, for some reason, you're not delivering quickly, securely, and reliably? Your process might need to change. Are you going to stop working on new features? I'd hope not.

You should take a more conservative approach initially, but as you progress, know there's always an opportunity to improve. The architecture will evolve, features will be removed, technology will need to be upgraded, etc. So as the Agile Manifesto says, you should always respond to change over following a plan. Don't expect that you'll need X time to be DevOps. It's a continuous improvement process.

What Worked for Others Might Not Work for You

In conclusion, your own DevOps path might be easy. Or it might make you suffer a bit. But just as organizations are different, cultures and processes can be different---even in the same industry. So why bother with sexy-but-useless rules that not only may not apply to you but also might hold you back? Just keep learning from others, taking what you think will work for you, and then trying things out. Pinky and the Brain were trying to take over the world every night, and they never did it.

It's important you're aware that DevOps trends that are sexy but useless exist. But here's some things that will help you.

Remove silos instead of adding more of them.
Know you don't need a certain tool or technology to provide value to your customers. Some will make your life easier and will improve productivity, but they alone won't solve all your problems.
Make sure every person in your organization knows how they are providing value. Doing so will reduce complaints about lack of access.
And most importantly, know that it's helpful to take the approach of always improving in small batches.

Experiment, adjust, and don't forget to enjoy the journey.

Growing a High-Performance DevOps Culture

Christian Melendez — Tue, 10 Dec 2019 15:56:24 +0000

Culture is one of those things where we all know what it is but can't explain it. Well, according to Wikipedia, culture is "the social behavior and norms found in human societies." But in simple words, it's all about people: how they interact, how they behave, how they talk, and what they practice. And culture is the foundation of a successful implementation of DevOps.

John Willis, an established speaker and writer on the subject of DevOps, coined the term CAMS (culture, automation, measurement, sharing) at a talk where he explained that DevOps culture is about breaking down silos. But what I find most striking about his discussion of culture, as summarized in the DevOps Dictionary, is the observation that "fostering a safe environment for innovation and productivity is a key challenge for leadership and directly opposes our tribal managerial instincts." So the starting point for your DevOps journey is good leadership. After that, it's just about how to grow your team to become a high-performing one.

A high-performing team in DevOps, according to recent research, is one that

Does deployments often, meaning several times a day.
Delivers a change with a fast lead time (minutes) after it's been pushed to a shared repository.
Has a short (again, minutes) mean time to recover (MTTR).
Has a small change failure rate (described here).

So how do you grow a high-performance DevOps culture? You create a culture that will produce a team that delivers on time with confidence in a predictable manner. Here are the things that will help you get there.

Increase Deployment Frequency

Increasing deployment frequency could be seen as a contradiction to delivering in a reliable manner. But working in small batches isn't anything new. Scrum and Extreme Programming are two management systems for software development that emphasize delivering in short cycles. While deployment frequency varies from one user to the next, people who practice Scrum typically deploy biweekly. And sure, that may seem fast if you're new to agile. But there are some companies that would find it laughable to call two weeks a "short cycle."

High performers are known for doing several deployments per day. The average high-performance team does four daily deployments, and some do way more than that. Etsy does eighty, and Amazon and Netflix deploy thousands of times every day, according to this study. That seems crazy, but doing so increases delivery throughput and lets organizations get feedback all the time. Customers or stakeholders don't have to wait for the sprint to finish. As soon as development has finished coding and verified that the change adds value (and doesn't break anything), there's no reason to keep waiting to deploy.

Working in small batches is more than a practice. It's a mindset. It requires that all members of the organization—developers, operators, businesspeople, and most importantly, leaders—think about it all the time. The result will be that you'll increase customer collaboration because they'll have something to try every day.

When you incubate an idea for too long, it becomes obsolete and useless. Deliver, and do it often.

Find Countermeasures to Increase Speed

How will you know if something's working or not if you don't measure it?

Measuring plays a big role in building a high-performance team. It's important that you identify what's stopping you from delivering on time. High performers have all sort of metrics in place to help them do that.

One very important metric is lead time. Lead time is how long it takes to implement a change in a production environment after the code has been committed in version control. Basically, it's how much time it takes to deploy and test code changes. This is important because it won't matter how fast the developer codes; a change could take weeks to be delivered.

Do you know what's causing big lead times in your workflow? You might not.

For example, you might be working with long-lived feature branches. Are you measuring the time it takes to merge the feature into the master branch? If you're spending too much time on this task that doesn't have much value, it's time to find a countermeasure.

According to lean training company Velaction, countermeasures are "the actions taken to reduce or eliminate the root causes of problems that are preventing you from reaching your goals." So a good countermeasure in the above example is to use trunk-base development. You'll always commit to the master branch, reducing the issue of having to spend too much time fixing merge conflicts.

After you've applied a countermeasure, you might need to apply another one somewhere else. But it will be hard to identify which if you're not measuring everything you're doing.

Having a culture of continuous learning in your organization will help you become better across the board. You won't stop improving once you've made the transition from doing deployments once a month to two weeks. You'll continue working to identify what's preventing you from doing daily deployments. And deployments are just one example. What about code coverage? Manual testing? Immutable infrastructure? The list goes on. But the idea is that you'll always have a chance to improve what remains.

Stop Pointing Fingers

There's nothing more harmful in a culture than pointing fingers when something goes wrong. No one that provoked a downtime said, "I'm bored. Let's bring the system down, just for fun." (Well, unless they're hackers.) And if colleagues make someone feel guilty after causing a downtime, will that person admit fault the next time? Not likely.

Failure is inevitable, so organizations should embrace it. It's better to find mechanisms to reduce the time it takes to recover from failure than to try to avoid mistakes. One way to avoid failure is to do deployments less frequently, but that stops innovation.

What you need in your organization is a blameless culture.

This is a culture that accepts that we all make mistakes sometimes. Being able to say, "I screwed up. I'm sorry. Can someone help me fix it?" will help your organization focus on a solution rather than the problem. And if there's no stigma associated with guilt, the team can be honest about who else contributed to the problem.

It's important to build trust in your team. Making mistakes shouldn't be a punishable offense. Instead, you should anticipate failure and then fail on purpose. Netflix does it all the time—they bring servers down just to test their fault-tolerant architecture.

Now, telling management that you want to intentionally cause problems doesn't sound good. But what if you did it in an environment other than production? What if it only affected certain users? These are ways that you can enable experimentation and build trust in the team.

Let your people learn from their mistakes. Find ways to prevent those things from happening again. When something bad happens, it's not about one person. Stop pointing fingers so everyone accepts responsibility for their mistakes. A healthy organization embraces failure and learns from it.

Enable Production-Like Environments

High performers have a low failure rate. Deployments don't usually fail because high-performing teams deploy in small batches, they have practices in place that reduce lead time, and most importantly, they have a way to recover quickly in case of any failure.

How is that possible?

Well, one option is to have a production-like environment. It lets everyone experiment with what it's like to work on a production server. And it lets the team practice deployments as if they were doing it in a production environment.

For example, if you're using Jenkins as your CI tool, you can make sure every developer has their own copy of Jenkins installed locally. Why? Having it running locally on their machines will let developers test their code before integrating it into the source control. The flow will be like this: do some changes, pull the latest changes from the repository, fix any conflicts, simulate the integration locally, and if it works, push the changes. By doing this, the practice of continuous integration will have the team always ready to deliver.

Work on building a culture that's always ready by practicing before going live. This includes building, packing, testing, deploying, rolling back...you name it. You'll avoid any surprises when going live, and that will reduce your failure rate, increasing the trust in the team.

Do It Again, but Better Every Time

You'll start seeing the benefits of everything I described here once you've made it a habit. After all, practice makes perfect. People might not see the value of these initiatives right away, but don't get frustrated. Just keep pushing until everyone gets it.

Fostering a safe environment for innovation and productivity should be the main goal in every DevOps implementation. Building trust in the team brings enormous benefits because members will have a feeling of ownership when doing things.

When you reward certain behaviors, you see that behavior more often. So make sure everything you reward contributes to building a high-performance culture.

DevOps Engineer: What Does It Take to Land the Job?

Christian Melendez — Tue, 03 Dec 2019 18:05:52 +0000

Every day, more and more companies are looking for people with DevOps knowledge. As I'm writing this post, there are 49,000 results on LinkedIn for DevOps jobs. That's a lot of job openings.

The high demand for DevOps engineers must be for a reason, right? Well, according to devops.com's "The State of DevOps Adoption and Trends in 2017" report, DevOps adoption has increased in the last couple of years, especially since 2016, and it won't stop. The problem is that some organizations are finding it hard to start the journey.

Although starting a DevOps initiative is not an easy task, looking for people that have the necessary skills is even harder. It doesn't matter what your background is; developer or sysadmin, you'll need to improve or acquire some new skills and knowledge to succeed in your day-to-day job.

Let's explore what you need to land a DevOps job.

Have Coding Skills

It's not necessary for you to have a developer background, although that will make it easier. But it is important that you feel comfortable reading code, that you understand it, and that you feel confident enough to change it or even create something from scratch.

You won't reinvent the wheel by creating a new software to do deployments, But you'll create scripts to automate deployments and other certain tasks.

So it's not important for you to know design patterns, object-oriented programming, functional programming, or a ton of languages. As long as you have developed your logic—meaning you have a developer's problem-solving mindset—reuse code rather than repeat yourself, and maintain other practices, you won't have any problems landing this job.

Skills are important, and you can develop them by learning just one programming language. The recommended (but not limited) list for you to choose from is Go, Python, Ruby, Bash, or PowerShell. There are various reasons why I'm recommending them. But essentially these are commonly-used languages. For example, you need to know Ruby if you want to use Chef. Having a developer background will make it easier for you to adapt to a team that's doing DevOps.

Domain-specific languages (DSL) are also very important here, depending on the tool you use. Some use JSON. Others use YAML. Those are representation formats, and that's how some tools interact. For example, when you define your infrastructure as code with tools like AWS CloudFormation or Terraform, you define that using YAML format. It's like you're programming, mostly in a structural or procedural way.

Know Version Control

It's normal for developers to be familiar with version control. But if your background is in operations, version control isn't that common in the field. At least, it wasn't until things like infrastructure as code came along.

The most important version control system is Git, so it's good to know what it is and how to use it: how to use branches, pull (or merge) requests, and more importantly, how to solve merge conflicts. You need to have a mindset of "if it's not in version control, I don't trust it."

A good way to practice version control is to use GitHub. There are good tutorials and courses out there that will help you to be confident using Git.

Know About Infrastructure

You need to know about infrastructure. And I don't mean in the sense of how to configure a Cisco router, although that's valuable. It's important that you know about networking, security, storage, and administering servers. You also need to familiarize yourself with key concepts like high availability, redundancy, self-healing, and resilient architectures—and not just in theory. You should also know how to apply them in real-world scenarios.

If you work for a startup, it's almost guaranteed that you'll be working with a cloud provider. So you need to know about cloud computing. The most popular ones right now are AWS, Azure, and Google. Pick one, get some experience, and if possible, get certified. Cloud certifications will improve your chances of getting the job and better pay.

If you work for an enterprise, you might be working with servers that are on-premises, in the cloud, or a combination of both. Some companies might plan to migrate everything to the cloud. So cloud is important, and its adoption is increasing every day in all types of organizations. But one thing's for sure: even if you don't work in a cloud environment, that doesn't mean that you can't adopt DevOps principles—especially when you know the critical practices.

So to work in DevOps, you need to know about continuous integration, continuous delivery, infrastructure as code, configuration management, log management, and immutable infrastructure. And when we talk about immutable infrastructure, we can't avoid talking about containers. Containers play a big role in infrastructure nowadays, especially Docker. Even if you don't wind up at an organization that's already using containers, you might find yourself at one that wants to start. And if there aren't plans for it, containers principles like packing, single responsibility, and immutable infrastructure can be applied too.

Know and Use Proper Tools

People in the industry say you should be “tool agnostic” or that you don't need to focus on tools. But in reality, it's good to be familiar with some tools. Choose a good set of tools and start doing something, anything—you’ll learn later if it was the right choice or not. Even though DevOps isn't just about tools, they're still important.

You'll find that many job postings require you to know tools like Jenkins, Chef, Puppet, Ansible, AWS CloudFormation, Docker, and Terraform. These are the commonly used ones. You need to know what problem each of those tools solves. Some are competent on their own; others are better if you combine them. When you start using these tools, you'll see how important it is that you know how to code.

Have Soft Skills

DevOps is all about people. As a DevOps engineer, you'll be spending a lot of time with developers and other teams.

You'll need to communicate the importance of applying some of the DevOps principles. That means knowing how to sell new ideas and having solid arguments for them. Initially, coaching the team will be important, especially if the company is just starting with DevOps. People are always a bit resistant to change. This is okay as long as your debates are healthy and constructive. And for that, it's required to have good communication skills, to be assertive, and to put yourself in the other person's shoes.

Developers also need to be aware of how things work and why some things are important: things like building once, feature flags, and continuous integration. These help deployments to be more reliable and predictable. The end goal should be to deliver quickly and frequently, with good quality and in a deterministic way.

Take the Challenge

In this post, we didn't deep dive into the topics. I just mentioned what you need to know to be a DevOps engineer. Already mastered everything mentioned here? Good. Keep improving!

But if there's a skill on this list you don't have, take the time to learn it and practice. You might get overwhelmed by everything you don't yet know, but relax. Start with just one thing. Tools will come and go, but they're getting better. And it's not just in how they solve problems; it's also in how they're creating well-written documentation so it's easy to learn.

Take the challenge because the demand for you as a DevOps engineer will only grow as time passes.

Why Are Engineers Getting DevOps Fatigue?

Christian Melendez — Tue, 19 Nov 2019 15:09:48 +0000

As an engineer, you already have enough responsibilities when developing software. Adding more tasks–say, DevOps-related ones—to your workday activities might not sound very appealing. With DevOps, not only are you responsible for producing working software, but now you also need to automate the building, testing, and deployment phases of the software. That's a lot to take care of! But the extra work aside, maybe you're just tired of the DevOps movement, and all the hype surrounding it is causing DevOps fatigue.

As a former developer, I can identify with that feeling of fatigue. I've also seen some colleagues reach a certain level of frustration with DevOps. There are times when we make the mistake of taking on everything, even the releases. This is especially common if we're perfectionists and don't like to deliver software with bugs. We could even get to the point of releasing our code to production. (Although now that you're "doing" DevOps, that might become your responsibility anyway.) After all, if we code it, we know the things that could go wrong and how to fix it if there are problems.

Even though now I'm more on the operations side of things—dealing with servers and helping companies implement DevOps—let me share some thoughts with you about why I think engineers are getting DevOps fatigue.

Working More but Getting the Same Pay

DevOps has stopped being perceived as only for developers and operators. But the downside of that is that some organizations are moving the DevOps functions to the developers. Developers might now feel they're being forced to take care of the delivery pipeline after they've finished their work. It's already hard enough for them to try to create software without bugs! More likely than not, developers are getting frustrated because they're making the same wages with more (and a different type of) work.

As I said before, there are times when developers feel the need to be doing more than they should. For example, if operations folks take too much time to answer their questions and requests, developers will always find a way to solve any problem themselves. They're even willing to learn new things like cloud and infrastructure-as-code. A lot of things happen when a developer finishes their code and it's ready to be shipped to production. At that point, it's not just about building and then copy/pasting artifacts. Deployments and releases could be more complicated, and one team can't handle everything.

I don't see developers going the extra mile as a bad thing. But there are times when these new responsibilities are being taken for granted. And that's where the problem with DevOps lies for engineers.

Experiencing Consequences From Losing Specialization

Even though engineers are still programming and DevOps fosters treating the infrastructure as code, engineers now need to learn about infrastructure. It's no easy task to provision and configure a server without wasting resources. Sure, software engineers need to make sure their code doesn't generate memory leaks and doesn't make excessive use of threads or networking. But the knowledge developers will need to handle those tasks is not enough to justify calling them experts on infrastructure.

Having to properly scale the application and operate the infrastructure could take some time to get right. There are so many things that operation folks need to consider, such as if servers are running on-prem or in a cloud or hybrid environment. It would be a lie to say that having a Chef recipe, Terraform template, or Ansible playbook is enough. It's better if the organization has a subject matter expert to make the most of the infrastructure and optimize costs. Nothing beats experience.

Again, it's not a bad thing that developers know more about infrastructure and how their code affects production systems. But some developers like creating application code. Developers might feel like DevOps is de-specializing them because of all the other things they need to take care of to release software.

DevOps practices and tools are continually growing and evolving, and the same goes for developers' fields. It's always good to work with subject matter experts because when those weird problems appear...who you gonna call? Certainly not the Ghostbusters.

Hearing Everyone Talk About DevOps

"Everyone is doing DevOps, and so should you." I'm sure you've heard the message.

I do believe that DevOps outcomes are great, but only when you've previously identified a problem where DevOps was a good fit for the solution. If you're just implementing DevOps because it's the new trend, you're probably just fine with your current delivery pipeline. It's not a good idea—especially with DevOps—to try to replicate what worked for others; everyone has different needs and problems.

I like the way Gene Kim has defined DevOps, but I'll only quote a sentence that's relevant to this post—I don't want to contribute to your fatigue. He said that "DevOps is not about what you do, but what your outcomes are." If DevOps is about what your outcomes are instead of what you do, it doesn't make any sense to adopt DevOps. You might be already doing some DevOps things and haven't noticed yet.

If you're just tired of all DevOps stuff, that's fine. I get that fatigue once in a while too. In my case, it's because people have an incorrect understanding of DevOps, which causes them to implement things that the DevOps movement was started to fix—things like creating another silo by having a separate team or adding the "full stack" title to everyone, which creates the idea that everyone needs to know about everything.

Getting More Pressure to Deliver Quickly

Another way that people are getting DevOps wrong is by setting the expectation that because they're "doing" DevOps now, the organization should be doing several deployments per day. It takes time to change things. DevOps implies a change in the culture, and changing a culture becomes harder when the organization is large. It's wrong to think that you'll be able to deliver faster overnight. There are so many things you need to take care of. Some examples are automated build and tests, production-like environments, configuration management, and so on.

There are a lot of unrealistic expectations with DevOps, and several deployments per day are one of them. Not surprisingly, many people started to hear about this when John Allspaw and Paul Hammond from Flickr talked about doing ten deployments per day at a conference. The 2017 state of DevOps report also reinforces the idea that high performers (companies like Amazon, Netflix, Google, and other "unicorns") do several deployments per day.

All of that is appealing, so I understand why some organizations would like to "do" DevOps. But to succeed you first need to identify where your problems are. There are tremendous benefits from doing several deployments per day, but it's not easy to get it right. I wrote a post on this a while back.

You shouldn't give yourself X months to be DevOps compliant and deliver faster. Engineers already have enough pressure to meet the deadlines, and adding more expectations doesn't do any good.

Outcomes Are What Matters

Let me repeat Gene's quote one more time. "DevOps is not about what you do, but what your outcomes are." When you embrace this idea, you start realizing that many of the things you think about DevOps don't make sense. Shifting activities to the left in your delivery pipeline doesn't mean that developers will now take care of everything or that operations folks will program new features in the system. Every piece in the puzzle has its purpose, and forcing people to add tasks outside of the scope of their duties to their workload is never a good practice.

Engineers might get tired with DevOps because they end up doing more work. Sure, they're learning new things, but they're also being de-specialized in their field. Or maybe management is adding more pressure with unrealistic expectations, not realizing that change doesn't happen overnight. But it's also possible that engineers have just had enough of hearing everyone talking about DevOps.

If you or your team is getting DevOps fatigue, forget about it for a moment. Identify where you're having problems when pushing your code changes to production. Reasons vary, but your process, tools, or even your people could be preventing you from releasing software effortlessly. The idea behind DevOps is not bad; the problem is that everyone has a different definition of DevOps and implements it in different ways, and that's frustrating.

DevOps vs. SRE? 4 Important Differences

Christian Melendez — Tue, 05 Nov 2019 16:46:51 +0000

If you happen to know what SRE is, you might be wondering how it relates to DevOps. Well, let's not beat around the bush. There's no "versus"—there's only a different approach for how to deliver better software faster. In this post, I'll break down each approach and show where DevOps and SRE differ. You'll notice that SRE has an opinionated approach for how to run production systems, whereas DevOps focuses more broadly on people, process, and tools—in that order of importance.

Let's start by setting the foundation for what DevOps and SRE are.

What's DevOps?

I'm not going to spend too much time on definitions, but I'll use them throughout the post to remark on the differences between DevOps and SRE. Of the many definitions of DevOps, I prefer this one from Gene Kim:

DevOps is [the] set of cultural norms and technology practices that [enables] the fast flow of planned work from, among others, development, through tests into operations while preserving world-class reliability, operation and security. DevOps isn't about what you do, but what your outcomes are.

So DevOps is mainly a cultural shift inside the organization, not a group, person, or position. What's essential in DevOps are the outcomes at the finish line. The "what" and "how" of it all isn't important. That's why the DevOps model CALMS outlines a set of principles that every DevOps initiative should consider using, and it starts with culture.

Now, let's continue with SRE.

What's SRE?

SRE stands for "site reliability engineering," a term coined by Google. A few years ago, Google engineers wrote a book to explain how they run and operate their systems in production. Then, they wrote a second book on practical ways to implement SRE. Both books are now available for free.

Google's definition of SRE is quite simple:

SRE is what happens when you ask a software engineer to design an operations team.

Therefore, SREs are operations folks with strong development backgrounds, and they apply engineering practices to solve common problems when running systems in production. SREs are responsible for making systems available, resilient, efficient, and compliant with the organization's policies (like change management).

Now, let's get into the details of where SRE differs from DevOps.

Removing Silos in the Organization

The DevOps movement was initiated to eliminate the silo between developers and operators. Developers want to deploy the features they just coded as soon as possible. Operations folks would like to slow down doing deployments to maintain available systems. How does DevOps solve this problem? Besides the CALMS framework, there are also principles from the three ways of DevOps, which aim to break down the silos between developers and operators. The DevOps Handbook best explains these three ways of DevOps and gives you a few ideas to apply.

SRE also removes silos. The difference is that instead of only finding ways to optimize flow between teams, SREs get their hands dirty. Being where the action is gives SREs better context for supporting systems in production. SREs integrate into the team as consultants, helping developers create more reliable systems. What's most important here is that SREs share ownership of running systems in production with developers.

Measuring a Successful Implementation

DevOps metrics focus mainly on how quickly and frequently deployments are happening and how often they go wrong. In other words, according to the 2017 report from Puppet and DORA (DevOps Research and Assessment), the key metrics in DevOps are the number of deployments, the lead time from code commit to releasing, the number of deployments failed, and how much time it takes to recover from failure. Feedback loops help DevOps continuously improve the quality of systems, and they open the door to experimentation.

SRE also depends on metrics to improve systems, but from the reliability perspective. The foundations for SREs are the service-level objective (SLO), service-level indicator (SLI), and service-level agreement (SLA). Each of these metrics will show how reliable the system is. SREs use these metrics to determine if a release for a change in the system will go to production or not. In SRE, speed and quality are products of reliable systems, and SREs focus on those types of metrics.

Pursuing CI/CD Practices

DevOps is a huge advocate for automation; I'd say that after culture, automation is the second most crucial aspect. In DevOps, the message is to automate as much as possible and make the releases boring. Many activities happen after a developer commits the code, and most of these activities can—and should—be automated. For example, you can automate leveraging the application's build process after integrating everybody's work in code to a machine. Or you can automate the process of deploying application changes, which is—or should be—the same every time. DevOps pursues CI/CD to increase the velocity and quality of the systems.

SRE pursues CI/CD for a different reason: to reduce the cost of failure. For SREs, all the tedious and repetitive tasks that are common in operations—like deployments, application restarts, or backups—aren't appealing. For that reason, SREs reserve a certain amount of time (for example, Google reserves 50%) for reducing the operational work or toil. SRE uses the same practices from DevOps, such as canary releases, blue/green deployments, and infrastructure as code. But SRE does so with the purpose of doing other more appealing things, like evolving the architecture or implementing new technologies.

Accepting Failures

DevOps fosters a blameless culture because every time something goes wrong, it's a learning opportunity. Accepting that failures will continue to happen is the first step. Instead of putting too much effort into making systems completely fault-tolerant, a DevOps culture finds ways to tolerate fault. Netflix is the most prominent advocate of this culture, with its Simian Army. Netflix is continuously bringing part of its system down so that it's just regular business when a real fault comes. If a set of servers goes down in a zone, Netflix automates the process of recreating servers in a different zone. And they practice it in a production environment all the time.

SRE practices blameless postmortem every time a failure in the system happens. The idea of blameless postmortems is to identify what caused the fault, then find ways to avoid having the same failure happen again in the same way. SRE also accepts failures, but they put numbers to it—they call it the error budget. After defining the SLI, SLO, and SLA, SRE determines how much failure is acceptable (the budget), because it's expensive to be 100% available. And in some cases, it's not possible.

Therefore, SREs determine how long it would be acceptable for the system to be down. For example, say the site can be down for about 43 minutes every month, which means the uptime is 99.9%. If the system has been down more than the allowed budget that month, releases are paused until the next month.

DevOps and SRE Don't Compete With Each Other

I very much like the way Google relates SRE with DevOps by using the following phrase:

class SRE implements interface DevOps

SRE and DevOps don't compete with each other. SRE is the name Google chose to define the way they do DevOps before the term DevOps was coined. There are slight differences, but as it happens when a class implements an abstract class, the implementation might choose to overwrite or extend the base functionality.

I'd say the main difference is that DevOps is a culture that broadly defines a way of doing things. Maybe that's why there are too many definitions of DevOps and many case studies from companies of different sizes and industries. By contrast, SRE has an opinionated way of doing things, but that's because it was born when Google published their explanation of how they run systems in production.

My two cents? Study each movement, and pick the practices and principles that work for your organization today. Tomorrow? Well, tomorrow things will change, and you might need to adopt new principles from DevOps and SRE.

Kubernetes Tutorial: Learn the Basics and Get Started

Christian Melendez — Tue, 29 Oct 2019 13:55:19 +0000

It's been over five years since Google open-sourced the Kubernetes project. Even so, you might still be wondering what Kubernetes is and how you can get started using it. Well, you're in the right place! In this post, I'm going to explain the basics you need to know to get started with Kubernetes. And I won't just be throwing concepts at you—I'll give you real code examples that will help you get a better idea of why you might need to use Kubernetes if you're thinking about using containers.

Getting Started With Hands-On Exercises

Before we dive into the core concepts of Kubernetes, you're going to want to brew a pot of coffee.

Now, let's get started by setting up your local environment. I'm not sure about you, but I learn more quickly by doing rather than just reading or memorizing concepts. Kubernetes's learning curve is significant, and sometimes you can get overwhelmed with all the ecosystem that surrounds this popular container-orchestration system. But once you have a solid knowledge base, you'll see that Kubernetes only happens to have a lot of pieces that you need to put together to create something cool.

If you don't want to go through the process of installing Kubernetes right away, there's a playground you can use called Play with Kubernetes (PWK). I'd recommend you take the PWK approach only if installing Kubernetes becomes complicated with your current computer. There are a lot of options to install and use Kubernetes, but I'll focus only on the local environment.

Installing the Kubernetes Command Line

Before installing Kubernetes, you need to install the command line (CLI) tool called kubectl (pronounced: cube-cuttle). Kubectl is the CLI you'll use to interact with any Kubernetes cluster, not just the local one. If you don't want to have a local Kubernetes installation, but you want to operate a Kubernetes cluster, you'll only use kubectl.

Installing kubectl is very simple, and the instructions vary depending on your OS. It's supported for Linux, Windows, or Mac. You can follow the detailed instructions in the official docs page. Run the following command to verify the installation:

kubectl version

Now continue with the local installation of Kubernetes.

Installing Kubernetes Locally

You start by installing Docker. Kubernetes does support other container systems, but I think using Docker is the easiest way to learn Kubernetes.

Depending on your OS, I'd recommend a specific install method:

If Linux, the preferred method is to install minikube. You'll need a hypervisor like VirtualBox or KVM.
If Windows or Mac, the preferred method is to install the latest version of Docker for desktop. You might need to enable Kubernetes if you are already using the newest version.

The installation process might take a while, so take this time to grab another cup of coffee.

Verify that Kubernetes is running with the following command:

kubectl get svc

If you don't have an error, that's it! You now have Kubernetes running locally. If you want to have a graphic interface, you can install it very quickly. In this guide, I'll only use the CLI, but it's good to have the GUI just in case you want to have a graphic view of all the components.

Kubernetes Core Concepts

You interact with Kubernetes using the API (typically via the kubectl CLI) with the purpose of setting up the desired state of your applications. That means that you tell Kubernetes how your application should run. Kubernetes will make sure that the application is running in the state you specify, even if there was a failure in the cluster or the load increase and you need to scale out the resources.

The way you set the desired state is by creating objects in Kubernetes. We'll take a look at some of the core objects that you'll need to deploy with zero-downtime a resilient application. You define these objects in a YAML format, and that's what I'll use to practice the concepts.

Pods

A pod is the smallest compute unit object you create in Kubernetes, and it's how you group one or more containers. You might know that containers have only one responsibility. But the reason why a pod has more than one container is for performance and co-scheduling purposes. Containers in a pod share the same networking, the same storage, and are scheduled in the same host.

For example, a web app could be a container that writes the logs to a storage device. Another container that collects logs in files could use the same storage device as the web app container. That way, you don't have more than one responsibility into a container. So you should think of a pod as if it was a representation of a virtual machine, but it isn't.

You create a pod object with a YAML file definition named pod.yaml, like the following:

apiVersion: v1
kind: Pod
metadata:
    name: helloworld
    labels:
        app: helloworld
spec:
    containers:
    - name: helloworld
      image: christianhxc/helloworld:1.0
      ports:
      - containerPort: 80
      resources:
        requests:
            cpu: 50m
        limits:
            cpu: 100m

Now you create the pod by running the following command:

kubectl apply -f pod.yaml

You then verify that the pod is running with this command:

$ kubectl get pods
NAME         READY     STATUS    RESTARTS   AGE
helloworld   1/1       Running   0          41s

If for some reason the pod dies, you don't have to tell Kubernetes to create the pod again. Kubernetes will recreate one automatically because the pod's state is not the same as the one you defined. Let's take a look at the following object then.

Deployments

Pods are mortal, and that means that your application is not resilient. A deployment object is how you can give immortality to pods. You define how many pods you want to have running all the time, the scaling policies, and the policy for zero-downtime deployments. If a pod dies, Kubernetes will spin up a new pod. Kubernetes continually verifies that the desired state matches with the object's definition.

Let's create a deployment.yaml file to say that we want to have two pods running all the time, like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
  labels:
    app: helloworld
spec:
  replicas: 3
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: christianhxc/helloworld:1.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 50m
          limits:
            cpu: 100m

Now let's tell Kubernetes that we want to create the deployment object by running the following command:

$ kubectl apply -f deployment.yaml
deployment.apps/helloworld created

Verify that the deployment exists by running the following command:

$ kubectl get deployments
NAME         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
helloworld   3         3         3            3           40s

You'll see that now you have three pods running plus the one we created previously.

$ kubectl get pods
NAME                          READY     STATUS    RESTARTS   AGE
helloworld                    1/1       Running   0          8h
helloworld-75d8567b94-7lmvc   1/1       Running   0          24s
helloworld-75d8567b94-n7spd   1/1       Running   0          24s
helloworld-75d8567b94-z9tjq   1/1       Running   0          24s

If you want to try killing a pod, run the command "kubectl delete podhelloworld-75d8567b94-7lmvc". You'll see how Kubernetes creates a new pod so that the desired state matches with the deployment definition. Every time you need to update the de container image, you'll change the deployment.yaml file and run the "apply" command. Then, Kubernetes will create new pods with the updated container image version. Until they're running, the pods with the previous version will be terminated. You can configure in much more detail how you want this update to happen, but for now the default behavior works for learning purposes.

Services

At the moment, you can't access the pods through a web browser to verify that they're working—you need to expose them. The recommended way to expose pods is via the service object. Still, other pods in the cluster can communicate between each other by using the internal IP address. But you can't rely on an IP to communicate with pods for its dynamism, which is why the solution is still a service.

A service then is a representation of a load balancer for pods. You can configure the load balancing algorithm, and if Kubernetes is integrated with a cloud provider, you'll use the native load balancers from the cloud provider.

The most straightforward way to define your services is as the following service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: helloworld

You might notice that we've been using labels and selectors, and that's because that's how a service knows to which pods redirect the requests. In the objects you've been creating, the label chosen is "app: helloworld" in the pod's metadata definition. It doesn't matter how a pod was created. As long as it has that label, it will be "registered" in the service.

Let's run the following command to create the service:

$ kubectl apply -f service.yaml
service/helloworld created

To verify that the service is running, and to get the IP address to test it, run the following command:

$ kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
helloworld   LoadBalancer   10.105.139.72   localhost     80:30299/TCP   21s
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        12h

Use the value given for "EXTERNAL-IP" and the "PORT" to test it. In this case, the URL should be http://localhost/api/values for the container image we're using. You might receive different values depending on where you're running Kubernetes. If for some reason port 80 is taken, you can change the port in the service.yaml file and run the "apply" command again.

And that's it! Just by using pods, deployments, and services, you can deploy a resilient stateless app in your local environment.

What's Next?

The next step is to move forward by deploying the same app to the following environment, like the cloud. The process and flow are the same, at least with a simple app like the one I used in the examples.

Even though this guide is a little more advanced than a typical "hello world" app, the app is still straightforward. But that's not usually the reality in today's complex and distributed world. I just scratched the surface—Kubernetes has a lot more concepts. (For example, I didn't talk about other core concepts like volumes, namespaces, ingress, configmap, secrets, jobs, daemonset, and statefulset.) But the concepts I explained and demonstrated here should be a good foundation for your Kubernetes journey.

If you want to continue learning more, the official documentation is a pretty good source, and here are some useful links to start:

Get your hands-on experience by practicing; it's the best way you'll learn Kubernetes.