DEV Community: Sylvia

Containers and Kubernetes vs VMs vs Config Management

Sylvia — Tue, 21 Jan 2020 16:15:00 +0000

In today's DevOps-centered world, it's often easy to be taken in by the infrastructure solution of the hour—VMs for everyone! Or wait, now containers are where it's at! Who needs config management anyway?

Those of us that are primarily developers probably haven't had to think about infrastructure decisions as much as our friends on the operations side. Since we haven't had to make those decisions in the past, it can be hard to figure out what's out there and why you'd want to use things like VMs or containers. We need to consider what actual problem these solutions are trying to solve. So let's start simply.

Config Management, VMs, and Containers: What Are They?

Let's take a high-level look at what they are and how they differ from one another.

Config Management

Configuration management is a centralized management of your decentralized systems and servers. Many consider configuration management to be not much more than package management—installing, upgrading, patching, and removing packages on an operating system. But modern config management like Puppet and Ansible do a lot more for you. It turns your infrastructure setup and decisions into code, allowing you to version and automate your infrastructure builds.

Virtual Machines

A VM takes an existing machine and installs additional compartmentalized machines on it. The initial machine is the host machine; all VMs on the host are called guests. The host manages its guests using a virtual machine monitor (or hypervisor). This keeps the guest VMs isolated from each other. Each guest VM also requires its own operating system. So the more VMs you have on a particular machine, the more operating systems are running.

Containers/Kubernetes

Containers like Kubernetes are a type of virtualization that let you run an application and its dependencies in an isolated process. A container doesn't have its own OS. It shares the kernel with not only the host but also the neighboring containers.

What Problem Do They Solve?

So there must be a reason why we have these different solutions. Let's consider what they're trying to fix.

Problem Solved By Config Management

Without config management, it would be difficult to track system-level changes and infrastructure. This problem becomes bigger the more servers you have. Modern configuration management is designed to deploy, configure, and manage servers in a repeatable and automated fashion.

At a container level, there's also the application's config management. Where do we store our configuration? For a container solution, Kubernetes uses config maps and Kubernetes secrets for management.

Problem Solved By Virtual Machines

VMs allow you to run different applications on the same hardware but in an isolated fashion. As a developer, you don't care what else was running on other VMs. You don't wonder about what was already using ports. And you're not worried about what particular OS versions or patches the other applications required. Therefore, if you have different applications that have different operating system needs, VMs will deal with that.

There have been several legacy systems that I've come across in the past that could benefit from VMs. They were inexplicably tied to not just a particular OS, but one particular version of that OS. It just would not run elsewhere. The secrets of why that was the case were often buried and long forgotten.

Problem Solved By Containers/Kubernetes

Containers let us run a single application in a lightweight fashion and still stay separate from other containers and applications on that server. This removes the requirement of having a separate VM for each application. All the containers use the same kernel as the host. Containers are also simpler for developers like us to spin up, as someone else already made the OS decisions. There's less for us to have to manage or even know about.

What Should You Watch Out For?

Consider the following factors when looking at config management, containers, and VMs.

Configurability

One of the benefits of config management is being able to patch and modify your infrastructure. You might think this would be a requirement since many companies still stand up their infrastructure setup by hand. However, your infrastructure is code. That should be versioned and automated through a CI/CD pipeline. If you have to change at a system level, then you can update your infrastructure scripts and it will deploy a new image. You want your application config to be immutable, and there's nothing keeping us from making the infrastructure immutable as well.

Kubernetes relies on config maps and Kubernetes secrets to ensure that containers have the correct configuration. That's where you set up your container so it knows what environment it's in and how it should run. It also makes secret rotation of things like database IDs and passwords easier; that can all be scripted and scheduled.

Another option for both VMs and containers is to have configuration completely in another system. The container or VM only knows where to get the configuration from. The configuration lives in a vault or KeyStore.

This has the advantage of being able to update the configuration for the whole fleet, but it makes the assumption that changing configuration won't introduce a bug. As much as we'd like to believe that, most of us have seen a change in configuration that caused problems. Perhaps configuration is better off being immutable.

Security

VMs don't have access to the host OS as they have their own OS to work with. Sneaking through the hypervisor to get through to the host OS is difficult. If Bob's Burger Barn VM is compromised, it's less likely to affect their neighbor over at Harriet's House of Hummus.

However, VMs are heavier solutions with their own set of OS and required packages. Therefore, they also have more places where you need to worry about penetration for security issues. With containers, they're considered to have a smaller surface area, as they only have the packages required to run the application.

The lifecycle of a container is also typically shorter. So if someone does break in, they can be shut out the next time you deploy the container.

On the other hand, containers share an OS and kernel. If someone gains kernel access in one container, they could have access to other containers as well. If a neighbor in your cluster is compromised, that could affect your application.

Additionally, security solutions are still evolving for containers. That's why you want to deploy frequently with the latest versions of tools and infrastructure using your config management. Even though containers have that smaller surface area that I mentioned, there aren't as many tools available yet for monitoring security risks and penetration. If a container is compromised, that container may be deleted and removed before we even find that something occurred.

Performance/Resources

Containers can have a performance lag over VMs. I once worked on a project moving from VMs to containers. Some of the components/applications that ran without issue on VMs, where the app was allocated 2–4G of RAM of memory, later required 6–8G of RAM on a container. Why is that?

For one, hypervisors on the VM boost performance so that it's closer to the performance of running on bare metal. Additionally, containers all run on the same kernel, similar to Linux processes. And if your application uses Java and runs on a JVM, there's another gotcha. Although Java 9 and 10 have some improvements, we were using Java 8. In that version, the JVM looked at the host OS to determine what processors were available, how many threads to spawn, and how much CPU it should use. That doesn't work when all containers are trying to use more than they have available.

Not all is perfect on the VM side, however. The overall efficiency of the host is worse if you're running multiple instances of an app on separate VMs than if you were using containers. As mentioned earlier, this is due to the overhead required for each VM and its own OS. That might not seem too bad, especially when you recall that VMs share the hardware of the host OS. However, the VM will also include "virtual" hardware that maps to the host the OS needs to run. It also comes with all the packages the OS needs to function fully.

In summary, If you want multiple instances of an application running, then containers make it easy and less resource-heavy. However, if you only have one instance of your applications running but have many applications overall, then VMs are a good solution.

Harmony

You may be thinking that none of these address all your problems or concerns. Or maybe one option would be attractive if it weren't for that single flaw. Well, you're in luck! You can use all three together to get all the benefits you're looking for.

For example, let's say that security of containers is a concern, but you still want to get your stuff easily deployed and configured. You can use VMs that hold containers to mitigate the risk of less secure neighbors while still having the flexibility and portability of containers. And since those VMs are all going to be using config management, setting this up is repeatable and configured correctly.

Config management isn't dead. It just needs to be adjusted for the current infrastructure. You'll always need to manage configuration. Even serverless applications, though they may not have dedicated servers, still live and run on a server. And someone has to manage those servers.

Overall, when combining these elements, you'll still have to monitor the system to make sure you're getting the most out of your infrastructure. Ensure it's working as expected with the help of tools for proper monitoring and logging.

Conclusion

VMs, containers, and configuration management solve specific problems. As developers, we can't make infrastructure decisions based on what's trendy. And we don't want to only have one tool in our tool belt.

Take all three of these solutions and see how they can address your problems without compromise.

GitHub Tips for Bootcamp Grads

Sylvia — Tue, 21 Jan 2020 00:30:27 +0000

I recently found myself at a Meetup where half of the attendees were from a local front-end bootcamp class. They were connecting, looking for future job prospects, and practicing their soft skills. They came off as energetic, engaged, and hopeful.

And somehow I found myself inspired to give advice that I’ve built up over the last decade or more of screening resumes and candidates. I hope they put the advice to good use. Now, I couldn’t give them all the tips I would like to. I would have bored them to tears with my talk. However, I want to share my thoughts with a wider audience.

I want to share these tips with you, the recent college or bootcamp grad, that doesn’t know how to stand out from his or her classmates that worked on exactly the same projects. It’s tough. And as bootcamp grads, or college grads, at least 70% of your GitHub repositories look exactly the same.

And that’s the problem. When you evaluate entry-level software developers, you need to find what makes one candidate stick out over another in just a couple of minutes. But when everyone does the same thing, it becomes more of a coin toss.

So what can you, an entry-level developer do? Do you need to spend months writing large projects or create amazing new applications? No. But you do need to take steps to improve your GitHub repos.

For today’s post, I’m going to take you through some examples of what I consider when reviewing your GitHub repo. These things may not be what all hiring managers or leads look for, but I know I’m not alone. And most of these suggestions won’t take months to implement.

Now let’s get started.

Do Your Repositories Stand Out?

Earlier I mentioned that 80% of repositories and GitHub profiles look the same. I made that number up based on my personal experience, but I don’t think I’m too far off the mark.

Don’t believe me? Just do a search for repos with something like “Lab 1” or “Lab 16” in the name. Then go to the owner’s profile and take a look around. Even though languages and problems vary, most of them look the same. So how can we make ours stand out? Let’s review the following tips.

1. Rename Your Repos

I get it, the instructors want your repos named with a lab or exercise number. However future employers don’t know what to do with this. We’ll maybe click in one or two of them to see if we can decipher what the code does and then give up and move onto another candidate.

For this, you have two options. First, you could append a meaningful name to the lab or exercise number. For example, instead of Lab 1, use Lab 1 – String Calculator.

Second, once your bootcamp class completes, feel free to rename your repos so that the name denotes what the program does.

Either way, you’re helping software developers that review your GitHub so we can drill into some of the more interesting examples.

2. Update Your README

Most lab examples don’t even include a README. However, in the real world README’s provide critical information on the usage of applications. Take a look at some examples of software libraries you’ve used to see what the README’s should contain.

For example, head on over to the Vue.js and see how much information gets put in there. As another example, head over to the Joda-Time GitHub repo for a shorter version of a README. Since Jada-Time is much smaller than Vue.js, less documentation is necessary.

Now you don’t need as much information as larger projects. But at the very minimum, add a short README and make it descriptive.

In other words, don’t create a README with just this:

Countries Application

But instead, do this:

An application that allows users to display lists of countries from a text file.

To build the application, clone the repository and run `mvn build`

To run the application, type mvn exec:java

3. Pin Your Best Work

When I look at your GitHub profile, your page displays a list of your most popular repositories. These may not be the projects you want your future employer to review.

Pick out your favorites and make sure they’re showing off your best work. Then pin them to your profile to make sure everyone sees what you want them to see on your profile.

4. Add Tests

Most companies now have some level of automated testing. If you want to stand out, show off some of your skills by adding a few unit tests to your repositories. It will let your future employers know that you’re at least familiar with basic testing concepts. Therefore, they know you’ll be able to hit the ground running faster than candidates that don’t use unit tests.

A word of caution here, however. I’ve seen many Java Spring projects where the developer used Spring’s initializer to create a basic template of the project. Part of that initial code includes a default contextLoads test. The only code this tests is whether your app can start or not. It does not give you any points.

You see, when I look at a project that has a test folder, I get my hopes. And I dive into it hoping to find something good. But I just end up seeing nothing but autogenerated code. Yuck!

Any amount of testing trumps no testing. Just put in a little effort and your repository will shine.

5. Use Configuration Properly

Another one of my pet peeves involves seeing code like this Java example:

private static final String JDBC_URL = "jdbc:postgresql://localhost:5432/my_awesome_db";
private static final String JDBC_USER = "myAwesomeUser";
private static final String JDBC_PASSWORD = "87654321";

There are actually a few problems here. First, these values don’t belong in your source code. They belong in configuration files or, in the case of the user id and password, in a system designed to store secrets.

At a minimum, pull those properties from a configuration file. Additionally, don’t ever store secrets in your GitHub repository. For the Java example above, let’s assume we’re using Spring. So we can add the following to our application.properties with placeholders for our secrets. Even if the user id and password aren’t real, treat them as if they were.

app.jdbc.url: "jdbc:postgresql://localhost:5432/web_lab_db"
app.jdbc.user: “<INSERT USER HERE>”
app.jdbc.password: “<INSERT PASSWORD HERE>”

This shows prospective employers that you understand what configuration does and that you understand at least a bit of security.

6. Use a Linter And Style Formatter

Code that follows a common practice looks cleaner and more professional. Additionally, using a linter ensures that you’re not making any common mistakes. Am I going to dig through your code looking for mistakes? No. But if you can reduce the amount of poor formatting and common mistakes, why not do it?

An additional note about formatting and style. Years ago I worked in an organization with a programmer that never indented things. Now, I understand that this is a really bad outlier, but it enforces my point. I struggled taking his code seriously. How can anyone appreciate his algorithmic knowledge when his code was unreadable?

7. Use .gitignore

Let’s look at a short list of folders and files that shouldn’t be committed in your git repository:

Operating system files: .DS_Store, Thumbs.db
IDE files like .idea
Language and framework files: jars, wars, and other executables
Directories of compiled code: target, bin, classes
Files from package managers like node_modules
Credentials and secrets Again, like other points on this list, knowing how to use your .gitignore properly will give you an edge over your competition.

8. Use folders and packages

A small percentage of repositories I review have no folders. They’re just repositories with 20 files and no organization. Unless there’s something else amazing about the candidate, these applications get thrown out.

Most types of applications use a standard folder structure. Look up what it should be for your language or framework and follow it.

Get A Leg Up On Your Competition

You’ve made it through my list. Congrats!

And you may have noticed that many of these items don’t have too much to do with the code itself. You’re an entry-level dev. Your code will not be as complex or performant as many applications. And that’s OK. We’re not looking for senior-level programming for entry-level positions.

But we still want to see something that stands out. Show your future employer a level of professionalism that doesn’t exist in many lab-type repositories. Show that you’re a professional that cares about standards and does more than just “make it work.”

Operational Visibility Is the Key to Your Competitiveness

Sylvia — Tue, 14 Jan 2020 15:40:36 +0000

With DevOps practices, many companies have improved their CI/CD pipeline, ensuring that deployments happen smoothly and frequently. In addition, many companies have added automated testing and application monitoring to make sure their apps work consistently and within spec.

Unfortunately, many companies stop there and don't take further actions to improve operational visibility. Additional tooling and infrastructure for metrics and logs don't always have clear benefits. Therefore, many companies deprioritize these efforts in favor of more features.

However, you're missing out on a lot of benefits if you just stop with the basics of DevOps. A successful deployment is only one step in the process. But if you stop there, you’re don't get the data that can provide more positive benefits for your customers. And more benefits for your customers can lead to better competitiveness in the market.

Let's look at operational visibility and see how it can give your company and products the edge over the competition.

Operational Visibility

Operational visibility refers to the monitoring of your system’s operations, performance, and readiness. It allows you to assess fluctuations in metrics and act on anomalies.

Many businesses continue to act with little operational visibility. They find out about problems from their customers. However, some problems never bubble up to the surface because affected users may simply go away without commenting on their experience.

So you shouldn't just stop where your CI/CD pipeline ends. But where do you go now? The next step involves adding operational visibility to your platform. Once you know your application made it into production, you'll want to know more than just how well it’s running. You also want to know what the system traffic looks like because that'll help you get into the minds of your customers and find out what they're actually doing with your products.

Competitiveness

Now that you know a bit more about operational visibility, how can it improve your competitiveness? In the next few sections, let's look at how data from our application and business operations can give you an edge.

The Features and Fixes That Customers Need

With increased operational visibility, you can track which features your customers gravitate toward in our applications and where they have problems or run into errors. This insightful additional data can drive your application teams to focus on the features that customers actually want and use.

Using all this data, you can plan our product launches so that you launch with the most critical bug fixes and most important features first.

Additionally, you can increase the speed with which you go to market with these features and fixes. Those additional metrics and insights will provide a better view of how your systems are used and will alert you to problems quickly if there's a need to backtrack.

Decentralized Decision-Making for a Faster Turnaround

With more visibility into the operations of your systems, your teams can make more decisions themselves. If a feature seems to be taking off quickly, perhaps a team can find a way to capitalize on the success right away instead of waiting until the next product meeting.

Decentralized decision-making brought about by increased operational visibility will provide a competitive edge where responsiveness and speed count.

Better Customer Support

Operational visibility provides the right data to the right people and also allows teams to be more autonomous. If your customer support folks can quickly answer questions by looking into metrics or logs of an application, they will route fewer calls down to product teams or developers. They'll be able to help customers within minutes instead of hours or days.

And with just a few of these competitive advantages, you can start separating yourself from similar products and applications that don't focus on daily operations and trends. But how do you add all this visibility into your world? Let's take a deeper look.

What Do You Need?

So let's say you're ready to take the plunge into operational visibility. What would your next steps be? Well, you need to work on a few things to make sure you're able to get what you need out of our systems and processes. Let's go through some of these steps.

Robust Data Collection

For the first step of operational visibility, you need data. And lots of it. Additionally, you need systems that can handle the load of storing and accessing all that data.

With Scalyr, the answer is simple. You can use logging for all your data needs and access it quickly. And it's not enough that you log all the things. You also need to make sure that everyone that needs access can have it.

Though it takes skill to write the code that runs your system, it doesn't take as much skill or experience to be able to read a log message and determine what's going on. That's especially true if your developers understand the value of the logs and add more user-friendly logging wherever they can.

So however you decide to gather your operational metrics and messages, make sure you're set up to handle the load on resources.

Alignment Between KPIs and the Data Collected

Whatever your business KPIs involve, there should be data collected that correlates to them. So if your product KPIs involve sales figures, conversions, or time spent on a page, then your operational visibility tool should also have access to that data as it relates to your products.

For example, it's no longer enough to have monthly or weekly reports on sales figures. You need these figures in real time. And if you gather those figures in real time using operational visibility tools, you can monitor the health of your company with better speed and accuracy.

So any time you're tracking KPIs for business, make sure you have visibility into the data that supports these KPIs.

Accurate Data

All the data that you're gathering and tracking towards KPIs doesn't add value if it's inaccurate. In order to make the best decisions around your competitiveness, you need to know that your data reflects reality.

So in addition to testing around your business logic, you need testing around your visibility metrics. You don't want to find out too late that a bug reports the wrong values for your KPI.

Visibility

Visibility can come in many forms. It can include fancy dashboards and alerting. Or it can involve providing the proper access to logs and metrics.

But the important thing about visibility involves actually making the data visible. And most companies stop short of providing the data to all the employees that can benefit from it.

For example, you may think that only developers and application maintenance folks need access to logs. But you're not considering that your customer support, social media, or marketing people could use access too.

Giving visibility to operational metrics can provide your employees with data that they can use in ways you can't even imagine. They can speed up support, find new ways to market features, or spot trends that you're not tracking.

Compete by Using Data

Instead of stopping with the easy wins brought by automated testing and a CI/CD pipeline, go further to build visibility into the operation of your applications and business.

You'll be able to react faster, improve support, and provide what other companies won't. And with all these features, your teams will benefit from the ability to make decisions autonomously with real-time data.

So take a look at Scalyr and give it a spin. See how you can make your product more competitive and successful using operational visibility.

Getting Started Quickly with Meteor Logging

Sylvia — Tue, 22 Oct 2019 15:29:31 +0000

We’ve covered a lot of bases in our "getting started quickly" posts. In addition to JavaScript and Node.js, we’ve created guides for C#, Java, Python, and more. Today we’ll discuss logging from a Meteor point of view.

Now, since Meteor can follow the same premise as JavaScript and Node.js, we’re going to change things up a bit. Instead of an empty app, we're going to clone an existing sample app and add logging to it. But if this tutorial isn't quite what you were looking for, go check out the basic JavaScript and Node.js posts. They'll cover anything else you might need.

But don’t worry, we’ll still cover these topics:

• Meteor overview
• Basic logging
• Where to log
• What to log

Now let's get started.

Meteor Overview

If you came to this blog post by searching for Meteor logging, you might not need this section. But I’d like to make sure we’re all starting from the same point, so here’s some quick Meteor info. Don’t worry, I’ll be brief.

Meteor provides a full-stack JavaScript platform for developing web and mobile apps. It includes a number of packages from Node.js and other JavaScript libraries. It focuses on and provides full stack reactivity so your UI acts seamlessly with your backend server. And as a bonus, it provides easy MongoDB integration right out of the box.

Basic Logging

As I mentioned above, instead of creating a bare application, we’re going to clone one of the existing sample applications that the folks over at Meteor have provided. Since we’re adding logging to an already existing app, we’ll be able to see different logs come through with minimal effort.

Setup

If you haven't used Meteor before, their basic setup is a breeze. First, you'll need to install both Node.js and npm. Also, if you're on a Mac, you'll need Xcode as well.

For step two, we'll install Meteor using either Chocolatey or a curl command, depending on your operating system.

curl https://install.meteor.com/ | sh

// OR...

choco install meteor

All set? Great. Next, let's clone the simple todo app from Meteor.

git clone https://github.com/meteor/simple-todos.git

Once you've got everything downloaded and installed, cd into the simple-todo directory and run the app using the Meteor command.

Once the application has started, go to http://localhost:3000/. You should see a simple todo app there.

You should be able to add users, add items to the checklist, and so on. Go ahead and play around with it. I'll be waiting at the next section for you.

Console Logging

Now that you've had a chance to check out the application, let's get started. To demonstrate some simple logging, we're going to add some console logs to our application. This can be helpful for debugging issues on the client side while you're developing. But it won't do much for you once your application is being used by someone out in the real world—you won't have access to their console logs.

To start, let's add some code to the imports/api/tasks.js file. This file contains the JavaScript that's executed when we add or remove tasks from the list. In the "tasks.insert" method, add the log.console line shown in this image.

Next, after saving that file, head over to your browser window. It should have updated automatically, so you don't have to restart the app. When you add a new task, you should see the output in your browser console.

Looks great, right? What if we also want to add some user context? Modify the line in the "tasks.insert" method to also print out the userId object that this application has defined.

console.log('Task being added: ' + text + ' by ' + this.userId);

After that, you should see something like this.

Not super helpful, is it? Let's make it a little more useful by asking Meteor to find the username associated with this ID instead.

console.log('Task being added: ' + text + ' by ' + Meteor.users.findOne(this.userId).username);

Yes, that looks much better!

OK, now that we've done some simple console logging, we can move on to the next section. However, I have one thing I'd like you to do first.

Now that you have a Meteor application up and running, take some time to explore its various functionalities by adding your own console logging throughout the code. It really helps you understand the application by seeing data flow through the system as you use it. And though these may not be logs you want to keep around forever, they'll help you gain a sense of what goes into a good Meteor application.

Where to Log

Let's return to something I mentioned at the beginning of the console logging section: You won't be able to view your customer's console.log files. Let's discuss what you should do instead.

Send Logs to the Server

As discussed, if your application is being used by actual customers, you won't have access to their console logs. Sure, you could have them open their browser console and send you what's there every time there's an issue. But you're assuming that people will know how to do that. And even if they do know, it doesn't provide a great user experience for them.

So what can we do instead? Well, instead of just writing to a console, we could send logs back to the server. But then what?

Save Logs for Later Use

With Meteor, one option could be to set up a logging table in MongoDB. So instead of just inserting tasks as our example above shows, we could insert logging or error messages. However, storing logs in a database typically doesn't make sense. You want to store them somewhere easily accessible by tools like Scalyr so you can start harvesting some analytics.

Let's consider another option. How about taking a look at some of Scalyr's APIs? There are a lot of goodies there. And two of them let us add events to our Scalyr logs, either one at a time or by batch. So instead of logging to the console, or saving the logs to our database, we can send events that require logging right back to Scalyr.

In this situation, I would send the logs over to the server side of my Meteor application and then call the Scalyr APIs with the token tied to my account. Let's go over this in greater detail.

First, adding simple HTTP functionality to your Meteor app requires running the following command within your Meteor directory.

meteor add http

That will let you make HTTP calls to remote servers easily.

Then we can use the following syntax to send an event to Scalyr's addEvents API.

HTTP.call('POST', 'https://www.scalyr.com/addEvents', 
{
  "token": "YOUR_ACCESS_TOKEN",
  "session": "UNIQUE_SESSION_ID",
  "sessionInfo": {
    "serverType": "simple-todo",
    "serverId": "simple-todo-server-1"},
  "events": [
    {
      "thread": "1",
      "ts": "1332851837424000000",
      "type": 0,
      "sev": 3,
      "attrs": {
        "message": "Task being added: stuff",
        "recordId": 39217,
        "latency": 19.4,
        "username": "sylvia"
      }
    }
  ]
);

You may have noticed that I broke out the username from the message I sent. That's because breaking parts of the log into separate attributes will help with searching later.

What to Log

With great logging comes great responsibility. So what should we log in our applications?

Well, that depends—not only on what your application does, but also on whether you're logging on the client side or the server side. You also want to make sure that what you log adds value. Logging every little action is usually a bad idea.

In fact, the example above doesn't represent something I'd want to be logged in my system. I don't care what my customers add to their to-do lists. I care about metrics, performance, and errors. And I want specific information that can help debug issues.

So what sort of data can we log?

Errors—If something goes wrong, we want to know about it.
Usage—Let's see how often various parts of our application are used. If we have features no one uses, we can deprecate them or make them better.
Performance—We want to know what the customer experiences regarding latency. If the application slows down too much for them, we need to investigate.

Specifically for error logging, what information do we need?

Timestamps that tell exactly when an error occurred
User ID or context, so we know who was affected
Relevant parameters to help recreate the problem and find a solution
Unique and specific error messages to help pinpoint the problem

You're on Your Way

You're in a great place. You've got a sample Meteor app to experiment with, as well as some guidance on how to log information in it. Once you've logged data from your application, remember to review the logs from time to time. Iterate on what's being logged and the format until you have a useful set of data that will help you debug and optimize your application. And now you can rest easy knowing you'll be able to support your Meteor application in production.

What Is a Site Reliability Engineer? Should You Become One?

Sylvia — Tue, 24 Sep 2019 14:20:11 +0000

Although site reliability engineering has been around for a while, it has only recently gained fame in general software circles. But there are still a lot of questions as to what a site reliability engineer (SRE) is and does.

Much of what we know comes from the book Site Reliability Engineering from Google. And we'll refer to that book a few times in this post.

SREs have been compared to operations groups, system admins, and more.

But the comparison falls short in encompassing their role in today's modern software environment.

SREs cover more responsibilities than operations. And though they usually have a background in system administration, they also bring software development skills to the role.

SREs combine all these skills and ensure that complex distributed systems run smoothly.

So how do they do all this?

Read further to find out what SREs are, in practice, and how they accomplish this through the responsibilities they fulfill.

What Is A Site Reliability Engineer?

Let's start our post with a definition. What is a site reliability engineer?

For starters, the concept of site reliability engineering was first introduced by a book of same name, written by a team of engineers from Google.

Wikipedia defines site reliability engineering like this:

Site Reliability Engineering (SRE) is a discipline that incorporates aspects of software engineering and applies them to infrastructure and operations problems.

In a nutshell, we can say that a SRE is a professional with solid background in coding/automation, that uses that experience to solve problems in infrastructure and operations.

Wait a minute...doesn't that sound a little bit familiar? What is the difference between SRE and DevOps?

First of all, you shouldn't think of DevOps as a role.

Rather, consider it more of a cultural aspect of the team. It can’t and shouldn't be assigned to a person, but rather performed as a team.

SRE, on the other hand, is the practice of creating and maintaining a service with high availability. The professional that assumes this role should be a software developer.

What Does a Site Reliability Engineer Do?

Automate All the Things

One difference between the SRE role and the traditional operations team involves automation.

In the past, operations folks would keep things running by executing scripts, pushing buttons, and carrying out other manual endeavors. However, in the SRE world, there's a heavy emphasis on automation.

Where did this drive come from? The engineering aspect of the SRE role.

When you put software developers in a position where the same functions repeat day in and day out, they'll be driven to automate. That's what software developers do best.

And automation doesn't stop at automating a software build and some acceptance tests. Their automation includes CI/CD and infrastructure creation and patching, as well as monitoring, alerting, and automating responses to certain incidents.

In Google's SRE book, this is also referred to as eliminating toil:

Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.

—Site Reliability Engineering

But why do we focus so much on reducing toil?

Not only does reducing toil make the processes repeatable and automated, but it also increases the amount of time SREs have to build tools and investigate infrastructure changes that further improve site reliability.

In summary, the less toil there is, the more time and resources are dedicated to making sure your software ecosystem runs reliably and the faster you can deliver business value.

Monitor Distributed Systems

With the popularity of distributed systems, there's a greater need for increased monitoring.

It's not enough that your application is up and running. We also need to ensure that infrastructure works properly and that all our other internal dependencies are accessible and functioning.

Additionally, business functions of the application should have proper monitoring to validate they're working properly.

For this portion of the job, SREs can use a product like Scalyr to monitor and alert on any potential issues. This allows them to monitor the system in real time as well as track long-term trends that may indicate reduced reliability.

Provide On-Call Support

Similar to traditional operations roles, SREs spend time rotating on-call responsibilities.

In addition to monitoring infrastructure and their own services, they also make themselves accessible to development teams for consultation and troubleshooting.

What does on call look like?

Typically SREs rotate the on-call role based on a schedule that allows other SREs to focus on engineering while also not causing burnout for the on-call engineer. On-call rotations vary from a couple days in a row to a week or more.

When a high-priority page triggers, the engineer will investigate and diagnose the issue. The SRE might also pull in additional engineers or software developers to resolve the issue.

Depending on the system's SLA, they may all need to work together to solve the issue in a matter of minutes. For low-priority issues, the SRE typically handles them during business hours.

And that's great news for the engineers who don't like to jump out of bed at 3 a.m. for every little thing that happens.

Log4j

Manage Incidents

An important part of the SRE role involves managing incidents.

Now you may say this is no different than the on-call responsibility. You find a problem and then fix it. How hard could it be?

Well, for managing incidents, SREs need to employ additional professional skills to make sure everything goes smoothly.

When an outage occurs, for example, there could be dozens of ways to diagnose and attempt to resolve the issue.

Therefore, to manage the incident properly, someone must monitor and facilitate the actions of all involved. And that requires clearly defined roles.

Though not all companies include these Google-recommended incident roles, we should at least consider them. These roles include the following:

An incident commander who maintains a high-level view of everything occurring.
The engineers who execute processes or modifications to the infrastructure or systems.
A communication role for relaying the right message to customers and management.
A planning role in charge of planning any meetings, handoffs, and logistical needs.
Without clearly defined roles for our SREs, we could have SREs that step on each other's toes as they try different solutions without up-front coordination and communication.

Facilitate Postmortems

Now that we've lived through an incident and resolved it in the sections above, we're ready for the postmortem.

Typically, an SRE facilitates or participates in these postmortems.

A postmortem brings together all relevant parties for analysis of the incident. The goal is to analyze what occurred during the incident and find the root cause. The participants also determine how the incident can be prevented or fixed in the future.

Some of the items that come out of a postmortem are listed below:

Stories to improve reliability or monitoring.
Additional documentation to assist with future incidents.
Further investigations or testing to prove out any hypothesis related to the incident.

Track Outages

Another responsibility of the SRE requires tracking outages. This eventually helps in identifying long-term trends and assists with creating reasonable SLOs and SLAs.

One use of tracking includes monitoring low-priority incidents. These incidents may not cause real issues for consumers, but looking at the long-term trends and timing can help isolate and resolve pesky bugs that don't seem to have a root cause.

Work With SRE and Development Teams

In addition to supporting development teams during on call, SREs also provide consulting and troubleshooting.

This assists both other SRE teams and software development teams that struggle with operational or reliability issues.

In this scenario, the SRE will assess current issues and determine which can be improved with automation or engineering effort. The SRE may also suggest solutions to reliability problems.

And perhaps most importantly, the SRE will drive changes to team processes. These changes will ensure that site reliability engineering enhances the team's ability to deliver value.

Create Service Level Indicators and Objectives

When you hear that a service has attained or is striving toward an uptime of 99.99 percent, you're talking about service level objectives (SLO). Service level indicators (SLI) measure these objectives.

In other words, the SLI is an agreement on how the SLO will be measured.

SREs assist with these by providing data for historical service performance. They also help provide realistic objectives for the future and might advise on proper SLAs for customers.

The SRE then works to make sure your application meets, though does not exceed, the stated SLO.

Now you may think that it's odd to not work to exceed an SLO. However, it would be a waste of resources to make something more reliable than it needs to be.

And SREs balance the needs of the customer with the goals of the services provided.

Responsibilities May Vary

In this post, we've discussed various activities that site reliability engineers participate in. Although these activities are done by many SREs, they aren't set in stone.

Companies do vary their SRE roles and responsibilities based on need. In general, companies that are at different points in the SRE journey may have different needs.

For example, a newer company may need SRE support in getting general outages under control. And most energy goes toward that base level of reliability.

However, other companies that are further along in the journey may have eliminated company-wide outages. They may spend more time on improving or validating service metrics that are business related.

For example, your pizza shop application may need new monitoring on its pizza recommendations once the site's general availability is stable and reliable.

Should You Become a SRE?

You should consider becoming a SRE if you have a solid foundation in automation.

Are you are a systems engineer who wants to improve your coding skills?
Are you are a developer who wants to learn how to manage large-scale systems?

If the answer for any of the questions above was "Yes," then this role is for you.

There is no better way to stay in touch with the most recent developments in the DevOps world. You'd also be able to increase your knowledge and skills in areas that are currently in high-demand.

Conclusion

As we discussed, SREs spend time on both technical and process-oriented responsibilities.

They do more than an operation or system administration team. They employ their engineering skills to automate and reduce the manual intervention necessary for administration tasks.

Additionally, they work with other engineering teams to provide proper monitoring, incident response, and management.

Over time, these functions improve the reliability and maintenance costs of your distributed systems.

And finally, they spread the culture of site reliability engineering through your organization so that all teams learn to make decisions with reliability in mind.