DEV Community: Hanif Jetha

The DigitalOcean Community Monthly: Redis & kubectl Cheat Sheets, Git for Writing, Hacktoberfest & More

Hanif Jetha — Tue, 01 Oct 2019 17:16:38 +0000

Welcome to The DOCOM Monthly, your monthly digest featuring some of the best content published in and around the DigitalOcean Community last month.

I'm Hanif Jetha, a DevOps Technical Writer on the Dev Education team. Today's roundup includes some helpful Redis and kubectl cheat sheets, a guide on using Git to manage writing projects, Hacktoberfest & more!

Without further ado, here are this month's featured posts:

How To Manage a Redis Database

This cheat sheet tutorial series by Mark Drake serves as a handy reference for common Redis operations like connecting to a Redis database, managing different data types, and troubleshooting and debugging problems.

How to Use Git to Manage Your Writing Project

Version control isn’t just for code. You can also use it to manage any sort of writing project. In this tutorial you’ll learn how to use Git to manage a small Markdown document. You’ll store an initial version, commit it, make changes, view the difference between those changes, and review the previous version. When you’re done, you’ll have a workflow you can apply to your own writing projects.

Getting Started with kubectl: A kubectl Cheat Sheet

Another cheat-sheet-style tutorial, this time for kubectl, the Kubernetes command-line interface. It covers common operations like configuring Contexts and Namespaces, rolling out Deployments, and fetching logs from running and terminated Pods. It also shows you how to install kubectl and set up shell autocompletion!

Featured Q&A

Get involved on the Community Q&A - ask questions and help other members by providing answers!

DigitalOcean on the Web

Michael Li teaches you how to Dockerize and deploy a machine learning web app in How to Deploy Your Machine Learning Web App to DigitalOcean.

DigitalOcean’s very own Erika Heidi demonstrates how to build a command-line PHP application in this multi-part series:

Bootstrapping a CLI PHP application in Vanilla PHP

Erika Heidi ・ Sep 20 '19 ・ 7 min read

#showdev #php #cli #beginners

Learn how to integrate DigitalOcean Kubernetes with GitLab and deploy your apps to a DOKS cluster: Kubernetes, GitLab and DigitalOcean.

Finally, DigitalOcean Developer Advocate Kamal Nasser shows you how to offload Docker image builds to a remote server:

How to Use a Remote Docker Server to Speed Up Your Workflow

Kamal Nasser ・ Sep 13 '19 ・ 5 min read

#docker

Community IRL

DigitalOcean’s Lisa Tagliaferri will be at the Grace Hopper Celebration in Orlando next week, delivering a talk on building inclusive and diverse open source communities. If you’re attending Grace Hopper, be sure to check out her talk!

Lisa Tagliaferri

@lisaironcutter

Thrilled to be speaking at #GHC19 next week 🎉

I'll be sharing tips for maintainers and contributors on expanding participation and promoting inclusivity & diversity in open source 👩🏿‍💻👩🏻‍💻👩🏽‍💻👩🏼‍💻

18:50 PM - 26 Sep 2019

6 17

Eddie Zaneski, a DigitalOcean Developer Relations manager, gave a talk at GitLab Commit on transforming a fresh Kubernetes cluster into a CI/CD bastion for GitLab. Check out his talk here:

Eddie will also be in attendance for the Hacktoberfest NYC Kickoff with DEV.

Hacktoberfest

Hacktoberfest is a month-long celebration of open source software (OSS) put together by DigitalOcean and DEV. Contribute four pull requests to open source projects on GitHub between October 1st and 31st, and receive a limited-edition Hacktoberfest T-shirt! You can register and learn more at the Official Hacktoberfest site, which also includes a list of IRL Hacktoberfest events you can attend.

Happy hacking, and see you all next month!

How to Set Up a Prometheus, Grafana and Alertmanager Monitoring Stack on DigitalOcean Kubernetes

Hanif Jetha — Mon, 29 Jul 2019 19:40:34 +0000

Introduction

Along with tracing and logging, monitoring and alerting are essential components of a Kubernetes observability stack. Setting up monitoring for your DigitalOcean Kubernetes cluster allows you to track your resource usage and analyze and debug application errors.

A monitoring system usually consists of a time-series database that houses metric data and a visualization layer. In addition, an alerting layer creates and manages alerts, handing them off to integrations and external services as necessary. Finally, one or more components generate or expose the metric data that will be stored, visualized, and processed for alerts by the stack.

One popular monitoring solution is the open-source Prometheus, Grafana, and Alertmanager stack, deployed alongside kube-state-metrics and node_exporter to expose cluster-level Kubernetes object metrics as well as machine-level metrics like CPU and memory usage.

Rolling out this monitoring stack on a Kubernetes cluster requires configuring individual components, manifests, Prometheus metrics, and Grafana dashboards, which can take some time. The DigitalOcean Kubernetes Cluster Monitoring Quickstart, released by the DigitalOcean Community Developer Education team, contains fully defined manifests for a Prometheus-Grafana-Alertmanager cluster monitoring stack, as well as a set of preconfigured alerts and Grafana dashboards. It can help you get up and running quickly, and forms a solid foundation from which to build your observability stack.

In this tutorial, we'll deploy this preconfigured stack on DigitalOcean Kubernetes, access the Prometheus, Grafana, and Alertmanager interfaces, and describe how to customize it.

Prerequisites

Before you begin, you'll need a DigitalOcean Kubernetes cluster available to you, and the following tools installed in your local development environment:

The kubectl command-line interface installed on your local machine and configured to connect to your cluster. You can read more about installing and configuring kubectl in its official documentation.
The git version control system installed on your local machine. To learn how to install git on Ubuntu 18.04, consult How To Install Git on Ubuntu 18.04.
The Coreutils base64 tool installed on your local machine. If you're using a Linux machine, this will most likely already be installed. If you're using OS X, you can use openssl base64, which comes installed by default.

Note: The Cluster Monitoring Quickstart has only been tested on DigitalOcean Kubernetes clusters. To use the Quickstart with other Kubernetes clusters, some modification to the manifest files may be necessary.

Step 1 — Cloning the GitHub Repository and Configuring Environment Variables

To start, clone the DigitalOcean Kubernetes Cluster Monitoring GitHub repository onto your local machine using git:

git clone git@github.com:do-community/doks-monitoring.git

Then, navigate into the repo:

cd doks-monitoring

You should see the following directory structure:

ls

Output
LICENSE
README.md
changes.txt
manifest

The manifest directory contains Kubernetes manifests for all of the monitoring stack components, including Service Accounts, Deployments, StatefulSets, ConfigMaps, etc. To learn more about these manifest files and how to configure them, skip ahead to Configuring the Monitoring Stack.

If you just want to get things up and running, begin by setting the APP_INSTANCE_NAME and NAMESPACE environment variables, which will be used to configure a unique name for the stack's components and configure the Namespace into which the stack will be deployed:

export APP_INSTANCE_NAME=sammy-cluster-monitoring
export NAMESPACE=default

In this tutorial, we set APP_INSTANCE_NAME to sammy-cluster-monitoring, which will prepend all of the monitoring stack Kubernetes object names. You should substitute in a unique descriptive prefix for your monitoring stack. We also set the Namespace to default. If you’d like to deploy the monitoring stack to a Namespace other than default, ensure that you first create it in your cluster:

kubectl create namespace "$NAMESPACE"

You should see the following output:

Output
namespace/sammy created

In this case, the NAMESPACE environment variable was set to sammy. Throughout the rest of the tutorial we'll assume that NAMESPACE has been set to default.

Now, use the base64 command to base64-encode a secure Grafana password. Be sure to substitute a password of your choosing for your_grafana_password:

export GRAFANA_GENERATED_PASSWORD="$(echo -n 'your_grafana_password' | base64)"

If you're using macOS, you can substitute the openssl base64 command which comes installed by default.

At this point, you've grabbed the stack's Kubernetes manifests and configured the required environment variables, so you're now ready to substitute the configured variables into the Kubernetes manifest files and create the stack in your Kubernetes cluster.

Step 2 — Creating the Monitoring Stack

The DigitalOcean Kubernetes Monitoring Quickstart repo contains manifests for the following monitoring, scraping, and visualization components:

Prometheus is a time series database and monitoring tool that works by polling metrics endpoints and scraping and processing the data exposed by these endpoints. It allows you to query this data using PromQL, a time series data query language. Prometheus will be deployed into the cluster as a StatefulSet with 2 replicas that uses Persistent Volumes with DigitalOcean Block Storage. In addition, a preconfigured set of Prometheus Alerts, Rules, and Jobs will be stored as a ConfigMap. To learn more about these, skip ahead to the Prometheus section of Configuring the Monitoring Stack.
Alertmanager, usually deployed alongside Prometheus, forms the alerting layer of the stack, handling alerts generated by Prometheus and deduplicating, grouping, and routing them to integrations like email or PagerDuty. Alertmanager will be installed as a StatefulSet with 2 replicas. To learn more about Alertmanager, consult Alerting from the Prometheus docs.
Grafana is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data. Grafana will be installed as a StatefulSet with one replica. In addition, a preconfigured set of Dashboards generated by kubernetes-mixin will be stored as a ConfigMap.
kube-state-metrics is an add-on agent that listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects like Deployments and Pods. These metrics are served as plaintext on HTTP endpoints and consumed by Prometheus. kube-state-metrics will be installed as an auto-scalable Deployment with one replica.
node-exporter, a Prometheus exporter that runs on cluster nodes and provides OS and hardware metrics like CPU and memory usage to Prometheus. These metrics are also served as plaintext on HTTP endpoints and consumed by Prometheus. node-exporter will be installed as a DaemonSet.

By default, along with scraping metrics generated by node-exporter, kube-state-metrics, and the other components listed above, Prometheus will be configured to scrape metrics from the following components:

kube-apiserver, the Kubernetes API server.
kubelet, the primary node agent that interacts with kube-apiserver to manage Pods and containers on a node.
cAdvisor, a node agent that discovers running containers and collects their CPU, memory, filesystem, and network usage metrics.

To learn more about configuring these components and Prometheus scraping jobs, skip ahead to Configuring the Monitoring Stack. We'll now substitute the environment variables defined in the previous step into the repo's manifest files, and concatenate the individual manifests into a single master file.

Begin by using awk and envsubst to fill in the APP_INSTANCE_NAME, NAMESPACE, and GRAFANA_GENERATED_PASSWORD variables in the repo's manifest files. After substituting in the variable values, the files will be combined and saved into a master manifest file called sammy-cluster-monitoring_manifest.yaml.

awk 'FNR==1 {print "---"}{print}' manifest/* \
 | envsubst '$APP_INSTANCE_NAME $NAMESPACE $GRAFANA_GENERATED_PASSWORD' \
 > "${APP_INSTANCE_NAME}_manifest.yaml"

You should consider storing this file in version control so that you can track changes to the monitoring stack and roll back to previous versions. If you do this, be sure to scrub the admin-password variable from the file so that you don't check your Grafana password into version control.

Now that you've generated the master manifest file, use kubectl apply -f to apply the manifest and create the stack in the Namespace you configured:

kubectl apply -f "${APP_INSTANCE_NAME}_manifest.yaml" --namespace "${NAMESPACE}"

You should see output similar to the following:

Output
serviceaccount/alertmanager created
configmap/sammy-cluster-monitoring-alertmanager-config created
service/sammy-cluster-monitoring-alertmanager-operated created
service/sammy-cluster-monitoring-alertmanager created

. . .

clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/sammy-cluster-monitoring-prometheus-config created
service/sammy-cluster-monitoring-prometheus created
statefulset.apps/sammy-cluster-monitoring-prometheus created

You can track the stack’s deployment progress using kubectl get all. Once all of the stack components are RUNNING, you can access the preconfigured Grafana dashboards through the Grafana web interface.

Step 3 — Accessing Grafana and Exploring Metrics Data

The Grafana Service manifest exposes Grafana as a ClusterIP Service, which means that it's only accessible via a cluster-internal IP address. To access Grafana outside of your Kubernetes cluster, you can either use kubectl patch to update the Service in-place to a public-facing type like NodePort or LoadBalancer, or kubectl port-forward to forward a local port to a Grafana Pod port. In this tutorial we'll forward ports, so you can skip ahead to Forwarding a Local Port to Access the Grafana Service. The following section on exposing Grafana externally is included for reference purposes.

Exposing the Grafana Service using a Load Balancer (optional)

If you'd like to create a DigitalOcean Load Balancer for Grafana with an external public IP, use kubectl patch to update the existing Grafana Service in-place to the LoadBalancer Service type:

kubectl patch svc "$APP_INSTANCE_NAME-grafana" \
  --namespace "$NAMESPACE" \
  -p '{"spec": {"type": "LoadBalancer"}}'

The kubectl patch command allows you to update Kubernetes objects in-place to make changes without having to re-deploy the objects. You can also modify the master manifest file directly, adding a type: LoadBalancer parameter to the Grafana Service spec. To learn more about kubectl patch and Kubernetes Service types, you can consult the Update API Objects in Place Using kubectl patch and Services resources in the official Kubernetes docs.

After running the above command, you should see the following:

Output
service/sammy-cluster-monitoring-grafana patched

It may take several minutes to create the Load Balancer and assign it a public IP. You can track its progress using the following command with the -w flag to watch for changes:

kubectl get service "$APP_INSTANCE_NAME-grafana" -w

Once the DigitalOcean Load Balancer has been created and assigned an external IP address, you can fetch its external IP using the following commands:

SERVICE_IP=$(kubectl get svc $APP_INSTANCE_NAME-grafana \
  --namespace $NAMESPACE \
  --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo "http://${SERVICE_IP}/"

You can now access the Grafana UI by navigating to http://SERVICE_IP/.

Forwarding a Local Port to Access the Grafana Service

If you don't want to expose the Grafana Service externally, you can also forward local port 3000 into the cluster directly to a Grafana Pod using kubectl port-forward.

kubectl port-forward --namespace ${NAMESPACE} ${APP_INSTANCE_NAME}-grafana-0 3000

You should see the following output:

Output
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

This will forward local port 3000 to containerPort 3000 of the Grafana Pod sammy-cluster-monitoring-grafana-0. To learn more about forwarding ports into a Kubernetes cluster, consult Use Port Forwarding to Access Applications in a Cluster.

Visit http://localhost:3000 in your web browser. You should see the following Grafana login page:

To log in, use the default username admin (if you haven't modified the admin-user parameter), and the password you configured in Step 1.

You'll be brought to the following Home Dashboard:

In the left-hand navigation bar, select the Dashboards button, then click on Manage:

You'll be brought to the following dashboard management interface, which lists the dashboards configured in the dashboards-configmap.yaml manifest:

These dashboards are generated by kubernetes-mixin, an open-source project that allows you to create a standardized set of cluster monitoring Grafana dashboards and Prometheus alerts. To learn more, consult the kubernetes-mixin GitHub repo.

Click in to the Kubernetes / Nodes dashboard, which visualizes CPU, memory, disk, and network usage for a given node:

Describing how to use these dashboards is outside of this tutorial’s scope, but you can consult the following resources to learn more:

To learn more about the USE method for analyzing a system's performance, you can consult Brendan Gregg's The Utilization Saturation and Errors (USE) Method page.
Google's SRE Book is another helpful resource, in particular Chapter 6: Monitoring Distributed Systems.
To learn how to build your own Grafana dashboards, check out Grafana's Getting Started page.

In the next step, we'll follow a similar process to connect to and explore the Prometheus monitoring system.

Step 4 — Accessing Prometheus and Alertmanager

To connect to the Prometheus Pods, we can use kubectl port-forward to forward a local port. If you’re done exploring Grafana, you can close the port-forward tunnel by hitting CTRL-C. Alternatively, you can open a new shell and create a new port-forward connection.

Begin by listing running Pods in the default namespace:

kubectl get pod -n default

You should see the following Pods:

Output
sammy-cluster-monitoring-alertmanager-0                      1/1     Running   0          17m
sammy-cluster-monitoring-alertmanager-1                      1/1     Running   0          15m
sammy-cluster-monitoring-grafana-0                           1/1     Running   0          16m
sammy-cluster-monitoring-kube-state-metrics-d68bb884-gmgxt   2/2     Running   0          16m
sammy-cluster-monitoring-node-exporter-7hvb7                 1/1     Running   0          16m
sammy-cluster-monitoring-node-exporter-c2rvj                 1/1     Running   0          16m
sammy-cluster-monitoring-node-exporter-w8j74                 1/1     Running   0          16m
sammy-cluster-monitoring-prometheus-0                        1/1     Running   0          16m
sammy-cluster-monitoring-prometheus-1                        1/1     Running   0          16m

We are going to forward local port 9090 to port 9090 of the sammy-cluster-monitoring-prometheus-0 Pod:

kubectl port-forward --namespace ${NAMESPACE} sammy-cluster-monitoring-prometheus-0 9090

You should see the following output:

Output
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

This indicates that local port 9090 is being forwarded successfully to the Prometheus Pod.

Visit http://localhost:9090 in your web browser. You should see the following Prometheus Graph page:

From here you can use PromQL, the Prometheus query language, to select and aggregate time series metrics stored in its database. To learn more about PromQL, consult Querying Prometheus from the official Prometheus docs.

In the Expression field, type kubelet_node_name and hit Execute. You should see a list of time series with the metric kubelet_node_name that reports the Nodes in your Kubernetes cluster. You can see which node generated the metric and which job scraped the metric in the metric labels:

Finally, in the top navigation bar, click on Status and then Targets to see the list of targets Prometheus has been configured to scrape. You should see a list of targets corresponding to the list of monitoring endpoints described at the beginning of Step 2.

To learn more about Prometheus and how to query your cluster metrics, consult the official Prometheus docs.

To connect to Alertmanager, which manages Alerts generated by Prometheus, we'll follow a similar process to what we used to connect to Prometheus. . In general, you can explore Alertmanager Alerts by clicking into Alerts in the Prometheus top navigation bar.

To connect to the Alertmanager Pods, we will once again use kubectl port-forward to forward a local port. If you’re done exploring Prometheus, you can close the port-forward tunnel by hitting CTRL-Cor open a new shell to create a new connection. .

We are going to forward local port 9093 to port 9093 of the sammy-cluster-monitoring-alertmanager-0 Pod:

kubectl port-forward --namespace ${NAMESPACE} sammy-cluster-monitoring-alertmanager-0 9093

You should see the following output:

Output
Forwarding from 127.0.0.1:9093 -> 9093
Forwarding from [::1]:9093 -> 9093

This indicates that local port 9093 is being forwarded successfully to an Alertmanager Pod.

Visit http://localhost:9093 in your web browser. You should see the following Alertmanager Alerts page:

From here, you can explore firing alerts and optionally silencing them. To learn more about Alertmanager, consult the official Alertmanager documentation.

In the next step, you'll learn how to optionally configure and scale some of the monitoring stack components.

Step 6 — Configuring the Monitoring Stack (optional)

The manifests included in the DigitalOcean Kubernetes Cluster Monitoring Quickstart repository can be modified to use different container images, different numbers of Pod replicas, different ports, and customized configuration files.

In this step, we'll provide a high-level overview of each manifest’s purpose, and then demonstrate how to scale Prometheus up to 3 replicas by modifying the master manifest file.

To begin, navigate into the manifests subdirectory in the repo, and list the directory’s contents:

cd manifest
ls

Output
alertmanager-0serviceaccount.yaml
alertmanager-configmap.yaml
alertmanager-operated-service.yaml
alertmanager-service.yaml
. . .
node-exporter-ds.yaml
prometheus-0serviceaccount.yaml
prometheus-configmap.yaml
prometheus-service.yaml
prometheus-statefulset.yaml

Here you'll find manifests for the different monitoring stack components. To learn more about specific parameters in the manifests, click into the links and consult the comments included throughout the YAML files:

Alertmanager

alertmanager-0serviceaccount.yaml: The Alertmanager Service Account, used to give the Alertmanager Pods a Kubernetes identity. To learn more about Service Accounts, consult Configure Service Accounts for Pods.
alertmanager-configmap.yaml: A ConfigMap containing a minimal Alertmanager configuration file, called alertmanager.yml. Configuring Alertmanager is beyond the scope of this tutorial, but you can learn more by consulting the Configuration section of the Alertmanager documentation.
alertmanager-operated-service.yaml: The Alertmanager mesh Service, which is used for routing requests between Alertmanager Pods in the current 2-replica high-availability configuration.
alertmanager-service.yaml: The Alertmanager web Service, which is used to access the Alertmanager web interface, which you may have done in the previous step.
alertmanager-statefulset.yaml: The Alertmanager StatefulSet, configured with 2 replicas.

Grafana

dashboards-configmap.yaml: A ConfigMap containing the preconfigured JSON Grafana monitoring dashboards. Generating a new set of dashboards and alerts from scratch goes beyond the scope of this tutorial, but to learn more you can consult the kubernetes-mixin GitHub repo.
grafana-0serviceaccount.yaml: The Grafana Service Account.
grafana-configmap.yaml: A ConfigMap containing a default set of minimal Grafana configuration files.
grafana-secret.yaml: A Kubernetes Secret containing the Grafana admin user and password. To learn more about Kubernetes Secrets, consult Secrets.
grafana-service.yaml: The manifest defining the Grafana Service.
grafana-statefulset.yaml: The Grafana StatefulSet, configured with 1 replica, which is not scalable. Scaling Grafana is beyond the scope of this tutorial. To learn how to create a highly available Grafana set up, you can consult How to setup Grafana for High Availability from the official Grafana docs.

kube-state-metrics

kube-state-metrics-0serviceaccount.yaml: The kube-state-metrics Service Account and ClusterRole. To learn more about ClusterRoles, consult Role and ClusterRole from the Kubernetes docs.
kube-state-metrics-deployment.yaml: The main kube-state-metrics Deployment manifest, configured with 1 dynamically scalable replica using addon-resizer.
kube-state-metrics-service.yaml: The Service exposing the kube-state-metrics Deployment.

node-exporter

node-exporter-0serviceaccount.yaml: The node-exporter Service Account.
node-exporter-ds.yaml: The node-exporter DaemonSet manifest. Since node-exporter is a DaemonSet, a node-exporter Pod runs on each Node in the cluster.

Prometheus

prometheus-0serviceaccount.yaml: The Prometheus Service Account, ClusterRole and ClusterRoleBinding.
prometheus-configmap.yaml: A ConfigMap that contains three configuration files:
- alerts.yaml: Contains a preconfigured set of alerts generated by kubernetes-mixin (which was also used to generate the Grafana dashboards). To learn more about configuring alerting rules, consult Alerting Rules from the Prometheus docs.
- prometheus.yaml: Prometheus's main configuration file. Prometheus has been preconfigured to scrape all the components listed at the beginning of Step 2. Configuring Prometheus goes beyond the scope of this article, but to learn more, you can consult Configuration from the official Prometheus docs.
- rules.yaml: A set of Prometheus recording rules that enable Prometheus to compute frequently needed or computationally expensive expressions, and save their results as a new set of time series. These are also generated by kubernetes-mixin, and configuring them goes beyond the scope of this article. To learn more, you can consult Recording Rules from the official Prometheus documentation.
prometheus-service.yaml: The Service that exposes the Prometheus StatefulSet.
prometheus-statefulset.yaml: The Prometheus StatefulSet, configured with 2 replicas. This parameter can be scaled depending on your needs.

Example: Scaling Prometheus

To demonstrate how to modify the monitoring stack, we'll scale the number of Prometheus replicas from 2 to 3.

Open the sammy-cluster-monitoring_manifest.yaml master manifest file using your editor of choice:

nano sammy-cluster-monitoring_manifest.yaml

Scroll down to the Prometheus StatefulSet section of the manifest:

Output
. . .
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: sammy-cluster-monitoring-prometheus
  labels: &Labels
    k8s-app: prometheus
    app.kubernetes.io/name: sammy-cluster-monitoring
    app.kubernetes.io/component: prometheus
spec:
  serviceName: "sammy-cluster-monitoring-prometheus"
  replicas: 2
  podManagementPolicy: "Parallel"
  updateStrategy:
    type: "RollingUpdate"
  selector:
    matchLabels: *Labels
  template:
    metadata:
      labels: *Labels
    spec:
. . .

Change the number of replicas from 2 to 3:

Output
. . .
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: sammy-cluster-monitoring-prometheus
  labels: &Labels
    k8s-app: prometheus
    app.kubernetes.io/name: sammy-cluster-monitoring
    app.kubernetes.io/component: prometheus
spec:
  serviceName: "sammy-cluster-monitoring-prometheus"
  replicas: 3
  podManagementPolicy: "Parallel"
  updateStrategy:
    type: "RollingUpdate"
  selector:
    matchLabels: *Labels
  template:
    metadata:
      labels: *Labels
    spec:
. . .

When you're done, save and close the file.

Apply the changes using kubectl apply -f:

kubectl apply -f sammy-cluster-monitoring_manifest.yaml --namespace default

You can track progress using kubectl get pods. Using this same technique, you can update many of the Kubernetes parameters and much of the configuration for this observability stack.

Conclusion

In this tutorial, you installed a Prometheus, Grafana, and Alertmanager monitoring stack into your DigitalOcean Kubernetes cluster with a standard set of dashboards, Prometheus rules, and alerts.

You may also choose to deploy this monitoring stack using the Helm Kubernetes package manager. To learn more, consult How to Set Up DigitalOcean Kubernetes Cluster Monitoring with Helm and Prometheus. One additional way to get this stack up and running is to use the DigitalOcean Marketplace Kubernetes Monitoring Stack solution, currently in beta.

The DigitalOcean Kubernetes Cluster Monitoring Quickstart repository is heavily based on and modified from Google Cloud Platform’s click-to-deploy Prometheus solution. A full manifest of modifications and changes from the original repository can be found in the Quickstart repo’s changes.md file.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Becoming a Technical Writer: The Paths 3 Engineers Took to their First Community Tutorial

Hanif Jetha — Mon, 25 Mar 2019 16:02:57 +0000

Regular readers of DigitalOcean Community tutorials (like How To Secure Apache with Let's Encrypt on Ubuntu 18.04 or How To Spin Up a Hadoop Cluster with DigitalOcean Droplets) may have noticed that beyond setup instructions, tutorials often contain insights, tips, and pain points surfaced from real-world production scenarios. This is because many of them are written by DevOps engineers, sysadmins, and software developers eager to share solutions and workarounds to problems faced while rolling out software and systems on the job.

Through tutorials, DigitalOcean writers contribute their in-the-trenches experience to the wider developer community, and in doing so solidify their own understanding of technical concepts. Some authors work full-time on the DigitalOcean Community writing team, while others contribute tutorials as part of the Write for DOnations program, which matches the author's payout with a charitable donation to a tech-focused nonprofit. All Community contributors share a common spirit of “giving back” to readers through teaching, whether these readers are seasoned engineering managers, or students wading in the waters of setting up an Nginx server or Jupyter Notebook for the first time.

DigitalOcean tutorials can broadly be categorized as either conceptual or procedural. An Introduction to Kubernetes, for example, is more conceptual in nature, as the author provides the reader with an overview of a piece of software or DevOps concept and boils it down to a set of digestible core ideas. How To Set Up an Elasticsearch, Fluentd, and Kibana (EFK) Logging Stack on Kubernetes, on the other hand, is a procedural walkthrough to support a developer setting up their infrastructure. The tutorial brings the reader through the installation and configuration of one or several technologies step-by-step, often providing valuable insight and context along the way.

Becoming a Technical Writer

There is no “one” path to becoming a DigitalOcean tutorial writer. Mitchell Anicas, formerly a senior technical writer on the Community team, and now a senior software engineer on the Billing team, began his career as a systems administrator at the University of Hawaii. After relocating to New York, he leveraged his years of experience administering systems and automating their configuration and deployment to begin writing Linux and infrastructure tutorials full-time.

“While working as a sysadmin, many of the tutorials I referenced weren’t complete, or weren’t really that high quality. It was a lot of someone writing blog posts saying something like ‘this is how this worked for me,’” he says. “Since I had been on the other side, I had a lot of empathy for readers.” Although he’d never written a tutorial before, with the help of other Community writers (all tutorials are peer-edited and tech-tested by another member of the Community team), he began publishing articles like How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04 and 5 Common Server Setups For Your Web Application.

On the other hand, Erika Heidi, a software engineer and writer based out of Amsterdam, had always been a writer. “I’ve always enjoyed writing, since I was very young,” she says. “I figured out that I could use blogging as a platform for documenting technical things like setting up servers and fixing common Linux problems, both as a future reference for myself and also as a way to share what I was learning.”

As a Community author, she’s contributed many tutorials, ranging from the procedural How To Secure Apache with Let's Encrypt on Ubuntu 16.04, to conceptual articles like What is High Availability and An Introduction to Configuration Management that draw from her extensive experience as a DevOps engineer. Like Mitchell, her technical experience endowed her with a keen sense for the problems her peers were facing and solutions she could provide to fill the gap.

Jeremy Morris followed a similar path to publishing his first DigitalOcean tutorial: “Throughout college as a computer science student, I would occasionally write blog posts about some of the things I learned at my internships, as a way of gaining a deeper understanding of the topics and sharing my knowledge with others,” he recalls. One of his professors, Lisa Tagliaferri, now managing the team of in-house Community writers at DigitalOcean, recommended he leverage his newly gained Python and Django experience and write a tutorial series on building a blog. This eventually led to him publishing Django tutorials like How To Install Django and Set Up a Development Environment on Ubuntu 16.04, How To Create a Django App and Connect it to a Database and How To Create Django Models, all topics he had become familiar with through his professional work.

Mitchell, Erika, and Jeremy all became writers at different stages of their careers in tech. Their common desire to help their peers through sharing hard-won solutions to challenging DevOps problems led them to publish their first DigitalOcean tutorials. While writing and educating others presented them with the need to develop distinct communication skills, they found that the challenge of describing solutions in a clear and accessible manner was well worth developing alongside their engineering experience.

Why Write?

Technical writing can be a powerful complement to engineering work, requiring the author to understand and process concepts at a deeper level than may be required to complete day-to-day tasks. Erika, in her drive to deeply understand the technologies she works with as a DevOps engineer, finds that writing tutorials helps her truly gain familiarity with new ideas: “There’s a lot of stuff in engineering I know how to do because I’ve done it multiple times before and it simply works, but if I had to explain how it works, I couldn’t,” she notes. “Writing helps me ‘untangle’ my thoughts, because I have to explain and organize those thoughts into logical steps.”

It can also be an incredibly rewarding pursuit. By publishing open-source tech tutorials, Erika feels that she is helping others who have similarly lent a hand throughout her learning path: “I believe the fulfillment comes from the feeling that I’m sharing something that might be useful for others. I’m searching for tutorials and how to do stuff all the time on the internet, and this is one way I can give back to the community.”

Mitchell shares this sense of being able to contribute his experience and knowledge through tutorials: “It’s cool to be able to help thousands of people through writing, especially writing tutorials for DigitalOcean, whose tutorials are respected and recognized by the developer community at large. I’ll be at a conference and run into someone and they’ll say something like ‘Oh yeah I used that tutorial of yours to set up an ELK stack, thanks so much!’”

Taking the Plunge and Diving into the Technical Writing

So how do you get started writing your first DevOps, software development, or systems tutorial? By jumping into the deep end, of course! Through the Write for DOnations program you can submit a short writing sample (for inspiration, take a look at this list of suggested topics) and work with our Community editors to have your article edited, tech-tested, proofread, and guided to publication. In addition, you'll be paid for your work, and DigitalOcean will match your payout with a donation to a tech-focused charity of your choice. To date, DigitalOcean has donated over $13,000 through external-author submissions and the Write for DOnations program!

In addition, the Community team frequently has openings for new writers, editors, and developer advocates to educate, curate, and produce some of the highest quality software-focused tutorials on the web. If you’re a DevOps or software engineer and have some experience writing documentation or other content, consult our Careers page for an up-to-date list of open full-time Community positions.

Although he may have been referring to something other than the pains of backing up and replicating a large distributed MySQL database, let Hemingway’s words guide you as you set sail on your technical writing voyage: “write hard and clear about what hurts.”

Modernizing Applications for Kubernetes

Hanif Jetha — Mon, 22 Oct 2018 16:13:14 +0000

Introduction

Modern stateless applications are built and designed to run in software containers like Docker, and be managed by container clusters like Kubernetes. They are developed using Cloud Native and Twelve Factor principles and patterns, to minimize manual intervention and maximize portability and redundancy. Migrating virtual-machine or bare metal-based applications into containers (known as "containerizing") and deploying them inside of clusters often involves significant shifts in how these apps are built, packaged, and delivered.

Building on Architecting Applications for Kubernetes, in this conceptual guide, we'll discuss high-level steps for modernizing your applications, with the end goal of running and managing them in a Kubernetes cluster. Although you can run stateful applications like databases on Kubernetes, this guide focuses on migrating and modernizing stateless applications, with persistent data offloaded to an external data store. Kubernetes provides advanced functionality for efficiently managing and scaling stateless applications, and we'll explore the application and infrastructure changes necessary for running scalable, observable, and portable apps on Kubernetes.

Preparing the Application for Migration

Before containerizing your application or writing Kubernetes Pod and Deployment configuration files, you should implement application-level changes to maximize your app's portability and observability in Kubernetes. Kubernetes is a highly automated environment that can automatically deploy and restart failing application containers, so it's important to build in the appropriate application logic to communicate with the container orchestrator and allow it to automatically scale your app as necessary.

Extract Configuration Data

One of the first application-level changes to implement is extracting application configuration from application code. Configuration consists of any information that varies across deployments and environments, like service endpoints, database addresses, credentials, and various parameters and options. For example, if you have two environments, say staging and production, and each contains a separate database, your application should not have the database endpoint and credentials explicitly declared in the code, but stored in a separate location, either as variables in the running environment, a local file, or external key-value store, from which the values are read into the app.

Hardcoding these parameters into your code poses a security risk as this config data often consists of sensitive information, which you then check in to your version control system. It also increases complexity as you now have to maintain multiple versions of your application, each consisting of the same core application logic, but varying slightly in configuration. As applications and their configuration data grow, hardcoding config into app code quickly becomes unwieldy.

By extracting configuration values from your application code, and instead ingesting them from the running environment or local files, your app becomes a generic, portable package that can be deployed into any environment, provided you supply it with accompanying configuration data. Container software like Docker and cluster software like Kubernetes have been designed around this paradigm, building in features for managing configuration data and injecting it into application containers. These features will be covered in more detail in the Containerizing and Kubernetes sections.

Here’s a quick example demonstrating how to externalize two config values DB_HOST and DB_USER from a simple Python Flask app’s code. We'll make them available in the app’s running environment as env vars, from which the app will read them:

hardcoded_config.py:

from flask import Flask

DB_HOST = 'mydb.mycloud.com'
DB_USER = 'sammy'

app = Flask( __name__ )

@app.route('/')
def print_config():
    output = 'DB_HOST: {} -- DB_USER: {}'.format(DB_HOST, DB_USER)
    return output

Running this simple app (consult the Flask Quickstart to learn how) and visiting its web endpoint will display a page containing these two config values.

Now, here’s the same example with the config values externalized to the app’s running environment:

env_config.py

import os

from flask import Flask

DB_HOST = os.environ.get('APP_DB_HOST')
DB_USER = os.environ.get('APP_DB_USER')

app = Flask( __name__ )

@app.route('/')
def print_config():
    output = 'DB_HOST: {} -- DB_USER: {}'.format(DB_HOST, DB_USER)
    return output

Before running the app, we set the necessary config variables in the local environment:


export APP_DB_HOST=mydb.mycloud.com

export APP_DB_USER=sammy

flask run

The displayed web page should contain the same text as in the first example, but the app’s config can now be modified independently of the application code. You can use a similar approach to read in config parameters from a local file.

In the next section we’ll discuss moving application state outside of containers.

Offload Application State

Cloud Native applications run in containers, and are dynamically orchestrated by cluster software like Kubernetes or Docker Swarm. A given app or service can be load balanced across multiple replicas, and any individual app container should be able to fail, with minimal or no disruption of service for clients. To enable this horizontal, redundant scaling, applications must be designed in a stateless fashion. This means that they respond to client requests without storing persistent client and application data locally, and at any point in time if the running app container is destroyed or restarted, critical data is not lost.

For example, if you are running an address book application and your app adds, removes and modifies contacts from an address book, the address book data store should be an external database or other data store, and the only data kept in container memory should be short-term in nature, and disposable without critical loss of information. Data that persists across user visits like sessions should also be moved to external data stores like Redis. Wherever possible, you should offload any state from your app to services like managed databases or caches.

For stateful applications that require a persistent data store (like a replicated MySQL database), Kubernetes builds in features for attaching persistent block storage volumes to containers and Pods. To ensure that a Pod can maintain state and access the same persistent volume after a restart, the StatefulSet workload must be used. StatefulSets are ideal for deploying databases and other long-running data stores to Kubernetes.

Stateless containers enable maximum portability and full use of available cloud resources, allowing the Kubernetes scheduler to quickly scale your app up and down and launch Pods wherever resources are available. If you don’t require the stability and ordering guarantees provided by the StatefulSet workload, you should use the Deployment workload to manage and scale and your applications.

To learn more about the design and architecture of stateless, Cloud Native microservices, consult our Kubernetes White Paper.

Implement Health Checks

In the Kubernetes model, the cluster control plane can be relied on to repair a broken application or service. It does this by checking the health of application Pods, and restarting or rescheduling unhealthy or unresponsive containers. By default, if your application container is running, Kubernetes sees your Pod as "healthy." In many cases this is a reliable indicator for the health of a running application. However, if your application is deadlocked and not performing any meaningful work, the app process and container will continue to run indefinitely, and by default Kubernetes will keep the stalled container alive.

To properly communicate application health to the Kubernetes control plane, you should implement custom application health checks that indicate when an application is both running and ready to receive traffic. The first type of health check is called a readiness probe , and lets Kubernetes know when your application is ready to receive traffic. The second type of check is called a liveness probe , and lets Kubernetes know when your application is healthy and running. The Kubelet Node agent can perform these probes on running Pods using 3 different methods:

HTTP: The Kubelet probe performs an HTTP GET request against an endpoint (like /health), and succeeds if the response status is between 200 and 399
Container Command: The Kubelet probe executes a command inside of the running container. If the exit code is 0, then the probe succeeds.
TCP: The Kubelet probe attempts to connect to your container on a specified port. If it can establish a TCP connection, then the probe succeeds.

You should choose the appropriate method depending on the running application(s), programming language, and framework. The readiness and liveness probes can both use the same probe method and perform the same check, but the inclusion of a readiness probe will ensure that the Pod doesn't receive traffic until the probe begins succeeding.

When planning and thinking about containerizing your application and running it on Kubernetes, you should allocate planning time for defining what "healthy" and "ready" mean for your particular application, and development time for implementing and testing the endpoints and/or check commands.

Here’s a minimal health endpoint for the Flask example referenced above:

env_config.py

. . .  
@app.route('/')
def print_config():
    output = 'DB_HOST: {} -- DB_USER: {}'.format(DB_HOST, DB_USER)
    return output

@app.route('/health')
def return_ok():
    return 'Ok!', 200

A Kubernetes liveness probe that checks this path would then look something like this:

pod_spec.yaml

. . .
  livenessProbe:
      httpGet:
        path: /health
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 2

The initialDelaySeconds field specifies that Kubernetes (specifically the Node Kubelet) should probe the /health endpoint after waiting 5 seconds, and periodSeconds tells the Kubelet to probe /health every 2 seconds.

To learn more about liveness and readiness probes, consult the Kubernetes documentation.

Instrument Code for Logging and Monitoring

When running your containerized application in an environment like Kubernetes, it's important to publish telemetry and logging data to monitor and debug your application's performance. Building in features to publish performance metrics like response duration and error rates will help you monitor your application and alert you when your application is unhealthy.

One tool you can use to monitor your services is Prometheus, an open-source systems monitoring and alerting toolkit, hosted by the Cloud Native Computing Foundation (CNCF). Prometheus provides several client libraries for instrumenting your code with various metric types to count events and their durations. For example, if you're using the Flask Python framework, you can use the Prometheus Python client to add decorators to your request processing functions to track the time spent processing requests. These metrics can then be scraped by Prometheus at an HTTP endpoint like /metrics.

A helpful method to use when designing your app's instrumentation is the RED method. It consists of the following three key request metrics:

Rate: The number of requests received by your application
Errors: The number of errors emitted by your application
Duration: The amount of time it takes your application to serve a response

This minimal set of metrics should give you enough data to alert on when your application's performance degrades. Implementing this instrumentation along with the health checks discussed above will allow you to quickly detect and recover from a failing application.

To learn more about signals to measure when monitoring your applications, consult Monitoring Distributed Systems from the Google Site Reliability Engineering book.

In addition to thinking about and designing features for publishing telemetry data, you should also plan how your application will log in a distributed cluster-based environment. You should ideally remove hardcoded configuration references to local log files and log directories, and instead log directly to stdout and stderr. You should treat logs as a continuous event stream, or sequence of time-ordered events. This output stream will then get captured by the container enveloping your application, from which it can be forwarded to a logging layer like the EFK (Elasticsearch, Fluentd, and Kibana) stack. Kubernetes provides a lot of flexibility in designing your logging architecture, which we'll explore in more detail below.

Build Administration Logic into API

Once your application is containerized and up and running in a cluster environment like Kubernetes, you may no longer have shell access to the container running your app. If you've implemented adequate health checking, logging, and monitoring, you can quickly be alerted on, and debug production issues, but taking action beyond restarting and redeploying containers may be difficult. For quick operational and maintenance fixes like flushing queues or clearing a cache, you should implement the appropriate API endpoints so that you can perform these operations without having to restart containers or exec into running containers and execute series of commands. Containers should be treated as immutable objects, and manual administration should be avoided in a production environment. If you must perform one-off administrative tasks, like clearing caches, you should expose this functionality via the API.

Summary

In these sections we’ve discussed application-level changes you may wish to implement before containerizing your application and moving it to Kubernetes. For a more in-depth walkthrough on building Cloud Native apps, consult Architecting Applications for Kubernetes.

We’ll now discuss some considerations to keep in mind when building containers for your apps.

Containerizing Your Application

Now that you've implemented app logic to maximize its portability and observability in a cloud-based environment, it's time to package your app inside of a container. For the purposes of this guide, we'll use Docker containers, but you should use whichever container implementation best suits your production needs.

Explicitly Declare Dependencies

Before creating a Dockerfile for your application, one of the first steps is taking stock of the software and operating system dependencies your application needs to run correctly. Dockerfiles allow you to explicitly version every piece of software installed into the image, and you should take advantage of this feature by explicitly declaring the parent image, software library, and programming language versions.

Avoid latest tags and unversioned packages as much as possible, as these can shift, potentially breaking your application. You may wish to create a private registry or private mirror of a public registry to exert more control over image versioning and to prevent upstream changes from unintentionally breaking your image builds.

To learn more about setting up a private image registry, consult Deploy a Registry Server from the Docker official documentation and the Registries section below.

Keep Image Sizes Small

When deploying and pulling container images, large images can significantly slow things down and add to your bandwidth costs. Packaging a minimal set of tools and application files into an image provides several benefits:

Reduce image sizes
Speed up image builds
Reduce container start lag
Speed up image transfer times
Improve security by reducing attack surface

Some steps you can consider when building your images:

Use a minimal base OS image like alpine or build from scratch instead of a fully featured OS like ubuntu
Clean up unnecessary files and artifacts after installing software
Use separate "build" and "runtime" containers to keep production application containers small
Ignore unnecessary build artifacts and files when copying in large directories

For a full guide on optimizing Docker containers, including many illustrative examples, consult Building Optimized Containers for Kubernetes.

Inject Configuration

Docker provides several helpful features for injecting configuration data into your app's running environment.

One option for doing this is specifying environment variables and their values in the Dockerfile using the ENV statement, so that configuration data is built-in to images:

Dockerfile

...
ENV MYSQL_USER=my_db_user
...

Your app can then parse these values from its running environment and configure its settings appropriately.

You can also pass in environment variables as parameters when starting a container using docker run and the -e flag:

docker run -e MYSQL_USER='my_db_user' IMAGE[:TAG]

Finally, you can use an env file, containing a list of environment variables and their values. To do this, create the file and use the --env-file parameter to pass it in to the command:

docker run --env-file var_list IMAGE[:TAG]

If you're modernizing your application to run it using a cluster manager like Kubernetes, you should further externalize your config from the image, and manage configuration using Kubernetes' built-in ConfigMap and Secrets objects. This allows you to separate configuration from image manifests, so that you can manage and version it separately from your application. To learn how to externalize configuration using ConfigMaps and Secrets, consult the ConfigMaps and Secrets section below.

Publish Image to a Registry

Once you've built your application images, to make them available to Kubernetes, you should upload them to a container image registry. Public registries like Docker Hub host the latest Docker images for popular open source projects like Node.js and nginx. Private registries allow you publish your internal application images, making them available to developers and infrastructure, but not the wider world.

You can deploy a private registry using your existing infrastructure (e.g. on top of cloud object storage), or optionally use one of several Docker registry products like Quay.io or paid Docker Hub plans. These registries can integrate with hosted version control services like GitHub so that when a Dockerfile is updated and pushed, the registry service will automatically pull the new Dockerfile, build the container image, and make the updated image available to your services.

To exert more control over the building and testing of your container images and their tagging and publishing, you can implement a continuous integration (CI) pipeline.

Implement a Build Pipeline

Building, testing, publishing and deploying your images into production manually can be error-prone and does not scale well. To manage builds and continuously publish containers containing your latest code changes to your image registry, you should use a build pipeline.

Most build pipelines perform the following core functions:

Watch source code repositories for changes
Run smoke and unit tests on modified code
Build container images containing modified code
Run further integration tests using built container images
If tests pass, tag and publish images to registry
(Optional, in continuous deployment setups) Update Kubernetes Deployments and roll out images to staging/production clusters

There are many paid continuous integration products that have built-in integrations with popular version control services like GitHub and image registries like Docker Hub. An alternative to these products is Jenkins, a free and open-source build automation server that can be configured to perform all of the functions described above. To learn how to set up a Jenkins continuous integration pipeline, consult How To Set Up Continuous Integration Pipelines in Jenkins on Ubuntu 16.04.

Implement Container Logging and Monitoring

When working with containers, it's important to think about the logging infrastructure you will use to manage and store logs for all your running and stopped containers. There are multiple container-level patterns you can use for logging, and also multiple Kubernetes-level patterns.

In Kubernetes, by default containers use the json-file Docker logging driver, which captures the stdout and stderr streams and writes them to JSON files on the Node where the container is running. Sometimes logging directly to stderr and stdout may not be enough for your application container, and you may want to pair the app container with a logging sidecar container in a Kubernetes Pod. This sidecar container can then pick up logs from the filesystem, a local socket, or the systemd journal, granting you a little more flexibility than simply using the stderr and stdout streams. This container can also do some processing and then stream enriched logs to stdout/stderr, or directly to a logging backend. To learn more about Kubernetes logging patterns, consult the Kubernetes logging and monitoring section of this tutorial.

How your application logs at the container level will depend on its complexity. For simple, single-purpose microservices, logging directly to stdout/stderr and letting Kubernetes pick up these streams is the recommended approach, as you can then leverage the kubectl logs command to access log streams from your Kubernetes-deployed containers.

Similar to logging, you should begin thinking about monitoring in a container and cluster-based environment. Docker provides the helpful docker stats command for grabbing standard metrics like CPU and memory usage for running containers on the host, and exposes even more metrics through the Remote REST API. Additionally, the open-source tool cAdvisor (installed on Kubernetes Nodes by default) provides more advanced functionality like historical metric collection, metric data export, and a helpful web UI for sorting through the data.

However, in a multi-node, multi-container production environment, more complex metrics stacks like Prometheus and Grafana may help organize and monitor your containers' performance data.

Summary

In these sections, we briefly discussed some best practices for building containers, setting up a CI/CD pipeline and image registry, as well as some considerations for increasing observability into your containers.

To learn more about optimizing containers for Kubernetes, consult Building Optimized Containers for Kubernetes.
To learn more about CI/CD, consult An Introduction to Continuous Integration, Delivery, and Deployment and An Introduction to CI/CD Best Practices.

In the next section, we’ll explore Kubernetes features that allow you to run and scale your containerized app in a cluster.

Deploying on Kubernetes

At this point, you’ve containerized your app and implemented logic to maximize its portability and observability in Cloud Native environments. We’ll now explore Kubernetes features that provide simple interfaces for managing and scaling your apps in a Kubernetes cluster.

Write Deployment and Pod Configuration Files

Once you've containerized your application and published it to a registry, you can now deploy it into a Kubernetes cluster using the Pod workload. The smallest deployable unit in a Kubernetes cluster is not a container but a Pod. Pods typically consist of an application container (like a containerized Flask web app), or an app container and any “sidecar” containers that perform some helper function like monitoring or logging. Containers in a Pod share storage resources, a network namespace, and port space. They can communicate with each other using localhost and can share data using mounted volumes. Addtionally, the Pod workload allows you to define Init Containers that run setup scripts or utilities before the main app container begins running.

Pods are typically rolled out using Deployments, which are Controllers defined by YAML files that declare a particular desired state. For example, an application state could be running three replicas of the Flask web app container and exposing port 8080. Once created, the control plane gradually brings the actual state of the cluster to match the desired state declared in the Deployment by scheduling containers onto Nodes as required. To scale the number of application replicas running in the cluster, say from 3 up to 5, you update the replicas field of the Deployment configuration file, and then kubectl apply the new configuration file. Using these configuration files, scaling and deployment operations can all be tracked and versioned using your existing source control services and integrations.

Here’s a sample Kubernetes Deployment configuration file for a Flask app:

flask_deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app
  labels:
    app: flask-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
    spec:
      containers:
      - name: flask
        image: sammy/flask_app:1.0
        ports:
        - containerPort: 8080

This Deployment launches 3 Pods that run a container called flask using the sammy/flask_app image (version 1.0) with port 8080 open. The Deployment is called flask-app.

To learn more about Kubernetes Pods and Deployments, consult the Pods and Deployments sections of the official Kubernetes documentation.

Configure Pod Storage

Kubernetes manages Pod storage using Volumes, Persistent Volumes (PVs) and Persistent Volume Claims (PVCs). Volumes are the Kubernetes abstraction used to manage Pod storage, and support most cloud provider block storage offerings, as well as local storage on the Nodes hosting the running Pods. To see a full list of supported Volume types, consult the Kubernetes documentation.

For example, if your Pod contains two NGINX containers that need to share data between them (say the first, called nginx serves web pages, and the second, called nginx-sync fetches the pages from an external location and updates the pages served by the nginx container), your Pod spec would look something like this (here we use the emptyDir Volume type):

pod_volume.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: nginx-web
      mountPath: /usr/share/nginx/html

  - name: nginx-sync
    image: nginx-sync
    volumeMounts:
    - name: nginx-web
      mountPath: /web-data

  volumes:
  - name: nginx-web
    emptyDir: {}

We use a volumeMount for each container, indicating that we'd like to mount the nginx-web volume containing the web page files at /usr/share/nginx/html in the nginx container and at /web-data in the nginx-sync container. We also define a volume called nginx-web of type emptyDir.

In a similar fashion, you can configure Pod storage using cloud block storage products by modifying the volume type from emptyDir to the relevant cloud storage volume type.

The lifecycle of a Volume is tied to the lifecycle of the Pod, but not to that of a container. If a container within a Pod dies, the Volume persists and the newly launched container will be able to mount the same Volume and access its data. When a Pod gets restarted or dies, so do its Volumes, although if the Volumes consist of cloud block storage, they will simply be unmounted with data still accessible by future Pods.

To preserve data across Pod restarts and updates, the PersistentVolume (PV) and PersistentVolumeClaim (PVC) objects must be used.

PersistentVolumes are abstractions representing pieces of persistent storage like cloud block storage volumes or NFS storage. They are created separately from PersistentVolumeClaims, which are demands for pieces of storage by developers. In their Pod configurations, developers request persistent storage using PVCs, which Kubernetes matches with available PV Volumes (if using cloud block storage, Kubernetes can dynamically create PersistentVolumes when PersistentVolumeClaims are created).

If your application requires one persistent volume per replica, which is the case with many databases, you should not use Deployments but use the StatefulSet controller, which is designed for apps that require stable network identifiers, stable persistent storage, and ordering guarantees. Deployments should be used for stateless applications, and if you define a PersistentVolumeClaim for use in a Deployment configuration, that PVC will be shared by all the Deployment's replicas.

To learn more about the StatefulSet controller, consult the Kubernetes documentation. To learn more about PersistentVolumes and PersistentVolume claims, consult the Kubernetes storage documentation.

Injecting Configuration Data with Kubernetes

Similar to Docker, Kubernetes provides the env and envFrom fields for setting environment variables in Pod configuration files. Here's a sample snippet from a Pod configuration file that sets the HOSTNAME environment variable in the running Pod to my_hostname :

sample_pod.yaml

...
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        env:
        - name: HOSTNAME
          value: my_hostname
...

This allows you to move configuration out of Dockerfiles and into Pod and Deployment configuration files. A key advantage of further externalizing configuration from your Dockerfiles is that you can now modify these Kubernetes workload configurations (say, by changing the HOSTNAME value to my_hostname_2) separately from your application container definitions. Once you modify the Pod configuration file, you can then redeploy the Pod using its new environment, while the underlying container image (defined via its Dockerfile) does not need to be rebuilt, tested, and pushed to a repository. You can also version these Pod and Deployment configurations separately from your Dockerfiles, allowing you to quickly detect breaking changes and further separate config issues from application bugs.

Kubernetes provides another construct for further externalizing and managing configuration data: ConfigMaps and Secrets.

ConfigMaps and Secrets

ConfigMaps allow you to save configuration data as objects that you then reference in your Pod and Deployment configuration files, so that you can avoid hardcoding configuration data and reuse it across Pods and Deployments.

Here's an example, using the Pod config from above. We'll first save the HOSTNAME environment variable as a ConfigMap, and then reference it in the Pod config:

kubectl create configmap hostname --from-literal=HOSTNAME=my_host_name

To reference it from the Pod configuration file, we use the the valueFrom and configMapKeyRef constructs:

sample_pod_configmap.yaml

...
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        env:
        - name: HOSTNAME
          valueFrom:
            configMapKeyRef:
              name: hostname
              key: HOSTNAME
...

So the HOSTNAME environment variable's value has been completely externalized from configuration files. We can then update these variables across all Deployments and Pods referencing them, and restart the Pods for the changes to take effect.

If your applications use configuration files, ConfigMaps additionally allow you to store these files as ConfigMap objects (using the --from-file flag), which you can then mount into containers as configuration files.

Secrets provide the same essential functionality as ConfigMaps, but should be used for sensitive data like database credentials as the values are base64-encoded.

To learn more about ConfigMaps and Secrets consult the Kubernetes documentation.

Create Services

Once you have your application up and running in Kubernetes, every Pod will be assigned an (internal) IP address, shared by its containers. If one of these Pods is removed or dies, newly started Pods will be assigned different IP addresses.

For long-running services that expose functionality to internal and/or external clients, you may wish to grant a set of Pods performing the same function (or Deployment) a stable IP address that load balances requests across its containers. You can do this using a Kubernetes Service.

Kubernetes Services have 4 types, specified by the type field in the Service configuration file:

ClusterIP: This is the default type, which grants the Service a stable internal IP accessible from anywhere inside of the cluster.
NodePort: This will expose your Service on each Node at a static port, between 30000-32767 by default. When a request hits a Node at its Node IP address and the NodePort for your service, the request will be load balanced and routed to the application containers for your service.
LoadBalancer: This will create a load balancer using your cloud provider's load balancing product, and configure a NodePort and ClusterIP for your Service to which external requests will be routed.
ExternalName: This Service type allows you to map a Kubernetes Service to a DNS record. It can be used for accessing external services from your Pods using Kubernetes DNS.

Note that creating a Service of type LoadBalancer for each Deployment running in your cluster will create a new cloud load balancer for each Service, which can become costly. To manage routing external requests to multiple services using a single load balancer, you can use an Ingress Controller. Ingress Controllers are beyond the scope of this article, but to learn more about them you can consult the Kubernetes documentation. A popular simple Ingress Controller is the NGINX Ingress Controller.

Here’s a simple Service configuration file for the Flask example used in the Pods and Deployments section of this guide:

flask_app_svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: flask-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: flask-app
  type: LoadBalancer

Here we choose to expose the flask-app Deployment using this flask-svc Service. We create a cloud load balancer to route traffic from load balancer port 80 to exposed container port 8080.

To learn more about Kubernetes Services, consult the Services section of the Kubernetes docs.

Logging and Monitoring

Parsing through individual container and Pod logs using kubectl logs and docker logs can get tedious as the number of running applications grows. To help you debug application or cluster issues, you should implement centralized logging. At a high level, this consists of agents running on all the worker nodes that process Pod log files and streams, enrich them with metadata, and forward the logs off to a backend like Elasticsearch. From there, log data can be visualized, filtered, and organized using a visualization tool like Kibana.

In the container-level logging section, we discussed the recommended Kubernetes approach of having applications in containers log to the stdout/stderr streams. We also briefly discussed logging sidecar containers that can grant you more flexibility when logging from your application. You could also run logging agents directly in your Pods that capture local log data and forward them directly to your logging backend. Each approach has its pros and cons, and resource utilization tradeoffs (for example, running a logging agent container inside of each Pod can become resource-intensive and quickly overwhelm your logging backend). To learn more about different logging architectures and their tradeoffs, consult the Kubernetes documentation.

In a standard setup, each Node runs a logging agent like Filebeat or Fluentd that picks up container logs created by Kubernetes. Recall that Kubernetes creates JSON log files for containers on the Node (in most installations these can be found at /var/lib/docker/containers/). These should be rotated using a tool like logrotate. The Node logging agent should be run as a DaemonSet Controller, a type of Kubernetes Workload that ensures that every Node runs a copy of the DaemonSet Pod. In this case the Pod would contain the logging agent and its configuration, which processes logs from files and directories mounted into the logging DaemonSet Pod.

Similar to the bottleneck in using kubectl logs to debug container issues, eventually you may need to consider a more robust option than simply using kubectl top and the Kubernetes Dashboard to monitor Pod resource usage on your cluster. Cluster and application-level monitoring can be set up using the Prometheus monitoring system and time-series database, and Grafana metrics dashboard. Prometheus works using a "pull" model, which scrapes HTTP endpoints (like /metrics/cadvisor on the Nodes, or the /metrics application REST API endpoints) periodically for metric data, which it then processes and stores. This data can then be analyzed and visualized using Grafana dashboard. Prometheus and Grafana can be launched into a Kubernetes cluster like any other Deployment and Service.

For added resiliency, you may wish to run your logging and monitoring infrastructure on a separate Kubernetes cluster, or using external logging and metrics services.

Conclusion

Migrating and modernizing an application so that it can efficiently run in a Kubernetes cluster often involves non-trivial amounts of planning and architecting of software and infrastructure changes. Once implemented, these changes allow service owners to continuously deploy new versions of their apps and easily scale them as necessary, with minimal amounts of manual intervention. Steps like externalizing configuration from your app, setting up proper logging and metrics publishing, and configuring health checks allow you to fully take advantage of the Cloud Native paradigm that Kubernetes has been designed around. By building portable containers and managing them using Kubernetes objects like Deployments and Services, you can fully use your available compute infrastructure and development resources.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License