DEV Community: 🌈 eric sorenson 🎹🔈🎚

Deployment Rollbacks via FireHydrant Runbook

🌈 eric sorenson 🎹🔈🎚 — Tue, 24 Nov 2020 00:00:00 +0000

FireHydrant has a sophisticated set of response actions for coordinating communications, activities, and retrospectives for incidents that affect your services. Relay helps by automating remediations that involve orchestrating actions across your infrastructure. In this example workflow, an incident that affects an application deployed on Kubernetes can trigger a rollback to a previous version automatically.

The workflow makes a couple of assumptions about your infrastructure that will need to be true to work out of the box; we’d love to work with you if you need additional flexibility! Specifically, it maps FireHydrant “Services” to Kubernetes deployments, and FireHydrant Environments to Kubernetes namespaces. Your deployment and rollback process is likely different from the one modelled here, but this should provide a good starting point for automating incident response activity with Relay.

Connecting the Services

In order to enable two-way communication between FireHydrant and Relay, we’ll need to do some work on both sides: Relay posting to FireHydrant requires a FireHydrant API key, and FireHydrant triggering Relay workflows uses a dynamically-generated Relay webhook URL.

First, add the Relay workflow to your account using this link. When you click “Save”, Relay will both create the webhook URL you’ll need and prompt that you’re missing a Secret and a Connection - we’ll get to those in a moment.

In FireHydrant, we’ll create a Runbook that will trigger the workflow by sending a webhook to Relay. Create a new Runbook and add a Send a Webhook step. For the Endpoint URL , paste the webhook address from the Relay’s Settings sidebar. The HMAC Secret field is an arbitrary string (not currently used). For the JSON Payload field, paste the following template:

{
  "incident_id": "{{ incident.id }}",
  "name": "{{ incident.name }}",
  "summary": "{{ incident.summary }}",
  "service": "{{ incident.services[0].name | downcase }}",
  "environment": "{{ incident.environments[0].name | downcase }}",
  "channel_id": "{{ incident.channel_id }}",
  "channel_name": "{{ incident.channel_name }}"
}

Next, create a FireHydrant API key for Relay to post information back into the incident timeline. Under Integrations - Bot users in FireHydrant, create a new Bot user with a memorable name and description. Save the resulting API token into a Relay secret on the Relay workflow’s Settings sidebar named apiKey (case-sensitive).

GCP Authentication Setup

This workflow uses a GCP Connection type on Relay’s end, which requires a service accountconfigured on your cluster. Follow the GCP guide to API Server Authentication’s “Service in other environments” section to set one up. This workflow will require the service account have the role roles/container.developer attached to it; if you re-use the connection for other workflows it may require additional permissions. Once you’ve gotten the service account JSON file downloaded, add a GCP Connection in Relay, name it relay-service-account and paste the contents of the JSON file into the dialog. Under the hood, Relay stores this securely in our Vault service and makes the contents available to workflow containers through the !Connection custom type in the workflow.

For non-GCP clusters, you can use Relay’s Kubernetes connection type, which requires less setup. GCP rotates access tokens every hour, making them unsuitable for automated use. The Kubernetes connection type needs an access token, the cluster URL, and the CA certificate for the cluster; there are more detailed instructions accompanying this deployment workflow example.

Configuring Services and Environments

One of the FireHydrant’s big benefits is its awareness of your infrastructure. It takes a little bit of up-front work, but if you invest the time to map out your services and environments, you can dramatically streamline your incident response. In this example, we’ve enumerated the microservices that make up our Sock Shop application and associated them with different runbooks. Fortunately for the demo, the sockshop-frontend service is a simple stateless Deployment in GKE, which makes new releases easy to manage with the kubectl rollout command.

Similarly, the Environments section lets you enumerate the instances of your service, to better characterize the impact of an incident, help assign owners for remediation actions, and message outage information to the appropriate audiences. Check out this FireHydrant helpdesk article on inventory management for more details on infrastructure organization. For our purposes, the goal of defining environments is to map them onto Kubernetes namespaces where our application is running. (For production workloads, it’s more likely that your environments map to distinct clusters; that’s totally possible to handle in Relay but is beyond the scope of this introduction!)

Incident Creation and Response

Now for the exciting part. Let’s say an update bumped the image on the frontend pods from a pinned version to latest and everything broke.

% kubectl set image deployment/sockshop-frontend nginx-1=nginx:latest \
   --record --namespace production deployment.apps/sockshop-frontend
image updated
% kubectl rollout history deployment sockshop-frontend --namespace production
deployment.apps/sockshop-frontend
REVISION CHANGE-CAUSE
1 kubectl set image deployment/sockshop-frontend nginx:1.18.0=nginx:latest --record=true --namespace=production
2 kubectl set image deployment/sockshop-frontend nginx-1=nginx:latest --record=true --namespace=production

But unbeknownst to the deployer, all was not well. After some troubleshooting, we determined that the application was degraded and rollout was to blame. In FireHydrant, the on-call person declares an incident and indicates the service and environment that were affected.

Once the incident is created, we can attach the rollback-via-relay Runbook that contains our webhook to the incident. The benefit of doing it this way is that the credentials are stored in Relay rather than needing a comamnd-line kubectl setup as above, and you don’t have to remember the exact syntax to type if something’s broken at 4AM! The correct steps are stored in the workflow, reducing the possibility for an error.

The incident timeline and associated Slack channel show these action taking place, and in Relay we can see the webhook come in, the workflow kick off, and ultimately post back into the timeline with the output of the rollback command. Thanks to FireHydrant’s awesome Slack integration, the updates roll into the channel in real time and chat messages are mirrored back into the incident so teams can coordinate their activities and keep a record of what happened. In this case, the rollback worked and we can resolve the issue quickly!

Conclusion and Next Steps

FireHydrant’s Runbook based system for coordinating actions in response to incidents is extremely powerful. The more you teach it about your infrastructure, the faster you’ll be able to respond when something goes wrong. And linking it to Relay via Runbooks enables another level of automated response and remediation. In this example, we were able to roll back a deployment without someone needing to run manual commands and potentially making things worse!

There are lots of existing Relay workflows that can act as building blocks or examples to construct your own incident response workflow. By combining them with clear processes codified in FireHydrant, responders can solve issues more quickly, reduce downtime, and get back to higher-value work.

To try this out for yourself, sign up for free FireHydrant account and Relay accounts, and get started automating!

Relay and Open Source

🌈 eric sorenson 🎹🔈🎚 — Tue, 23 Jun 2020 00:00:00 +0000

Relay and the open source community

Relay is deliberately built to be easy to extend. The more steps there are in the world the better it is for everyone who wants to write a workflow. It’s sort of like fax machines or MySpace: you’ll want to use the same technology that the people you’re communicating with use. OK, maybe the fax machine isn’t not a great analogy… the point is, nourishing a healthy open-source ecosystem and community around Relay is super important to its success.

In this post I’ll talk about the different aspects of Relay’s open source DNA: the ecosystem model, the codebase itself, and upstream projects.

The Relay Ecosystem

First and foremost, Relay is about workflows. Relay workflows are a well-defined YAML dialect that’s readily sharable. The first batch of workflows we’ve been writing to exercise the system and demonstrate its capabilities are on the Workflows section of the website with READMEs for example usage and a one-click button to add them to your account. The code for all of them is collected in the puppetlabs/relay-workflows repo. We’d love to see pull requests with bug fixes or improvements, new workflows you’ve written to solve problems with Relay, or requests for features. The workflow syntax is documented at relay.sh/docs along with instructions on how to set up VSCode to do automatic syntax validation and completion.

Relay workflows make use of containers to both respond to incoming events and to run as steps to accomplish the task you’re trying to automate. Step and Trigger container definitions that interact with a particular tool or service can be bundled up into what we call an Integration. The Integrations library on the website has a browsable, taxonomized list of the currently supported integrations.

We expect the current list will cover a lot of common Relay use cases, but there’s no way it’s going to cover all of them. If you run into the situation where there’s an integration with a service you use, but not a specific Step or Trigger that does what you want, we’d love to hear about it! Many of the features marked “Coming soon” on the integration pages just need some prioritization and tweaking to get out the door, and your vote counts to help get them on the list. Plus, all of the integrations are aggregated together under the relay-integrations Github organization, and by now it should go without saying that we welcome any PRs or issues on them.

To make a new integration, or if you want to go further than tweaking the existing containers, there’s a long-form Integrating with Relay guide that breaks down the different container types, the Python SDK, and best practices for integrating with the internal Relay service APIs.

The Relay Codebase

Speaking of documentation, the entire Relay documentation site is available on Github. If you run across anything in the docs that doesn’t seem right, from typos to examples that don’t work, we’d welcome corrections in the form of PRs or issues. Sending pull requests into the docs can be a great way to get involved with a project. Docs are generally on a quicker release cycle than the codebase and everybody benefits from having improved documentation.

If you’re a Go developer, whether aspirational (like me!) or expert, the Relay CLI tool could present a fun opportunity to dig into a relatively constrained codebase and improve the Relay development experience. I’ve written at length about our philosophy of CLI design, something I personally am pretty passionate about, and it’s been really gratifying to see those design principles come to life in the Relay CLI. There are a few issues tagged with CLI to get started, and of course we welcome your ideas too.

Going deeper into the architecture, the Kubernetes execution backend for the service is published at puppetlabs/relay-core. There are some intriguing possibilities for this codebase, like adding a local executor that could use minikube to run workflows locally. The metadata service would also be useful to have as a self-hosted version, because it would allow better validation and testing of step definitions while you’re developing workflows.

Of potential interest to front-end folks, the Puppet Design System is an open source collection of React components that the Relay web app (among other Puppet products) uses to provide gorgeous, modern look-and-feel.

Upstream

Relay makes heavy use of upstream open source technology. We try to be good citizens of these projects by contributing bug fixes and enhancements back upstream. This helps everybody because we don’t end up maintaining forks or carrying our own patches, the code gets more eyes on it, and the rest of the community can make use of the features too.

The projects we’re most heavily involved in are Tekton pipelines, which we’ve blogged about extensively and Ambassador, who posted a detailed technical blog post describing how Relay uses Knative and Ambassador together to run your Trigger containers.

Conclusion

To wrap things up, I’d like to acknowledge that open source is tough to get right. Communities are valuable, fragile things, which need lots of love and attention to flourish. We’re deeply committed to investing in community for Relay, but acknowledge up front that we won’t always get it right. We pledge to listen and improve based on your feedback. You can email the team at relay@puppet.com or jump on slack.puppet.com to chat in #relay. Please keep the lines of communication open and let’s build some awesome automation together!

3 Commandments for CLI Design

🌈 eric sorenson 🎹🔈🎚 — Wed, 08 Apr 2020 22:31:24 +0000

In the beginning…

In Neal Stephenson’s 1999 essay “In the Beginning Was The Command Line”, he describes how using Command-Line Interface (CLI) utilities is simpler and requires less code than a Graphical User Interface (GUI). They’re tiny, they do one thing really well, and they are meant to work in conjunction with other small, single-purpose tools to accomplish complicated tasks quickly.

However, the old-school Unix utilities are not without their problems. The passage of time and inevitable feature creep have led to situations like this:

usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The humble ls command now has more options than there are letters in the alphabet! Plus, despite the proliferation of features, many advanced and useful options are off by default to preserve backwards compatibility. Other core utilities are in similar situations: find and tar are stuck with arcane, user-hostile syntaxes, and the man pages for modern distributions are littered with comments like this one from the BSD-derived macOS 10.15 grep manual: “This implementation supports those options; however, their use is strongly discouraged.” (Discouraged? By whom? Why?)

Fast Forward

Relay.sh is an automation platform for event-driven devops workflows and we’re deeply committed to building awesome command-line experiences. I started writing down these principles as we began rethinking the command line tool for Relay. To me, the key UX principles that underpin the best of the new breed of projects in this vein are:

Enable Progressive Discovery — users are trying to solve a problem, but might start out knowing very little about the tool. The tool should guide them to a solution in iterative steps and provide plain-language help along the way. Users resorting to Google or StackOverflow is an anti-pattern here.
Go Upscale, Intelligently — the best tools infer the context they’re running in take advantage of modern capabilities if they can. For example, colorized text can greatly add to usability and aesthetics, but if the tool is being run in a pipeline, colors should be off to avoid polluting the output stream.
Be a Good CLI Citizen — there are a number of hard-fought CLI best practices that good tools follow, both to match users’ expectations and to peacefully co-exist with the rest of the ecosystem. It’s important for tool authors to be aware of these conventions, comply with as many as possible, and be deliberate about any significant divergence.

In the rest of the post I’ll provide some examples of each of these principles using CLI tools I’ve fallen in love with.

Enable Progressive Discovery

Pulumi is an infrastructure-as-code tool that allows developers to create and manage cloud infrastructure using general-purpose programming languages. While Pulumi programs can be written in Python, Typescript, .Net and so forth, the user interaction loop centers on using the pulumi CLI tool to interface between the local code repository and the Pulumi service. It’s a fantastic example of modern CLI design that exemplifies all of the principles I mentioned, but I’m going to focus on the way it enables progressive discovery.

First, installing it is a breeze because the website has helpful instructions for all the major OSes with a one-click copy-to-clipboard command to get you started.

Once you’ve got it installed, simply running the pulumi command with no arguments gives you helpful hints on what to do next in order to begin working with it. I’ve annotated the output of running pulumi with no arguments to illustrate this principle.

The output not only shows, right at the top, the first thing you ought to run if you’re new, it provides some hints as to what the second and third steps might be after you’ve begun. It links to the detailed docs on the website, invites exploration and discovery by giving the complete list of available commands, and has a reference section to show the flags which modify the program’s behavior in consistent ways, regardless of what subcommand you’re using.

Once you’ve gotten started and are working with a “stack” (a collection of resources that make up one application instance), the idea of progressive discovery shows up again. Running the pulumi stack subcommand without additional arguments doesn’t error out; instead it provides a reasonable guess as to what you meant: “show me information about the current stack”.

In addition to providing metadata about the stack and a nicely-formatted hierarchical representation of its resources, the output has links to the web app for more info and again shows hints about what to do next if this isn’t what you were looking for.

All in all, this experience is pretty delightful. It makes working with pulumi positive and enjoyable, and rather than feeling like a cut-down version of a web app it feels like a powerful first-class way of interacting with the service.

Go Upscale, Intelligently

My former colleague from Danger and all-around good human C J Silverio tweeted recently:

Ceej is sheltering under cats

@ceejbot

The rustlang ecosystem scene that’s making me happy right now: the retakes on ancient unix cli tools, with new approaches to info display.

bat
dua
exa

03:19 AM - 22 Feb 2020

3 30

This sent me down a rabbit hole of investigation and discovery. There’s indeed a vibrant community working in this space, and the ones CJ mentions are replacements for (respectively) cat, du, and ls. (There’s even a rewrite of the wc command from the Stephenson essay!) I came out the other side of the rabbit hole with a new set of aliases for my .zshrc:

# if rust stuff is found, use it
RBIN=$HOME/.cargo/bin
if [[ -d $RBIN ]]; then
  [[ -f $RBIN/bat ]] && for f in less more cat ; do 
     alias $f=bat ; 
  done
  [[ -f $RBIN/dua ]] && alias du=dua
  [[ -f $RBIN/rg  ]] && alias grep=rg
  [[ -f $RBIN/exa ]] && alias ls=exa
  [[ -f $RBIN/fd  ]] && alias fd=find
fi

These tools vary in their implementation but all of them exemplify the “Go Upscale” principle: they are aware of the affordances that modern terminal programs have and make use of them to provide upgraded experiences. To focus on one example, the lowly cat program is one of the oldest and most useful Unix utilities. Its name is short for “concatenate” and it can be used for all kinds of input and output stream manipulations, but it’s pretty primitive. bat (here’s its homepage on github) retains the usefulness of the original but adds colorized and line-numbered output, syntax highlighting for over a hundred file types, and cool features like git awareness. So bat README.md in my local hiera repository shows me:

The subtly shaded line numbers and line-drawn formatting, the + markers for uncommitted lines, and the helpful Markdown syntax highlighting are all super cool. But the thing that allowed me to alias cat=bat in my shell without fear of breaking scripts is that bat detects when it’s not printing to a terminal and turns all of that pretty-printing off so it retains compatibility.

Be a Good CLI Citizen

There’s a set of conventions and practices that high quality CLI tools share, which enable them to interact seamlessly with each other and, more importantly, allow a user to extrapolate lessons and habits learned from one tool into new contexts. This leads to a quicker sense of mastery and user delight because things feel “intuitive” (in reality, nothing about computers is truly intuitive to humans — there’s just experiences that confirm or confound our expectations based on previous encounters).

It’s tough to focus on a single tool as an example. Some of these conventions are defined in Jeff Dickey’s great post “12 Factor CLI Apps” about the Heroku CLI design philosophy, others from the GNU coreutils doc “Opening the software tool box”, and some are arguably matters of taste. Please comment if you think I’ve gotten something wrong or missed an important one!

Expect to operate as part of a pipeline and adjust behavior accordingly. Like the bat example above, turn off any fancy formatting if output isn’t a terminal so downstream tools see plain-text output for easy processing.
Don’t cross the streams! Closely related to pipeline operation, output should go to stdout and errors to stderr, so errors won’t be swallowed by redirection or intermingled with actual output.
Provide tabular and structured output formats for lists of data. For extracting only parts of a record, make it easy for users to split on whitespace to get just single fields. And no modern tool that outputs a lot of records in a table ought to be without a json output option.
For sophisticated CLIs, such as those which interact with remote services like Pulumi, think deeply about your command’s information architecture, then stick with it. A complex command invocation is like a grammatical sentence: it’s usually got a subject, a verb, and an object. For nontrivial commands there’s a hierarchy of these nouns and verbs that users put together to solve a problem. Either use a verb-object construction like kubectl get pods, where get is a top-level verb and the targets vary, or subject-verb construction like gcloud container clusters list (“clusters” is a subcommand of “container” and “list” is the verb).
Observe control conventions. Users frequently get things wrong, and sometimes that happens after they’ve hit the enter key on a command. The convention is that ^C (control-C) interrupts execution and exits the program, cleanly if possible. Similarly, ^Z (control-Z) suspends a program so that it can be paused or put in the background. Good CLI commands should honor these signals and behave predictably when they come in.
Don’t litter the filesystem. Use package managers and follow their conventions for the target operating system wherever possible. Similarly, follow conventions like XDG-Spec for guidance on where to install configuration files, docs, and supporting libraries. This kind of consistent packaging may take some additional work up front but it pays off in the long run when your software can be installed, upgraded, and config-managed alongside the rest of the user’s toolchain.

END OF FILE

When you’re done inputting a file into cat or its equivalents, a ^D (control-D) character indicates to the terminal that the end of the file’s been reached. Similarly, when ending a blog post it’s customary to summarize the points you’ve made and provide a stirring call to action for the reader. So, to recap the three top level design principles for modern command line interaction design:

Enable progressive discovery to lead users down a low-friction path to solving their problem with your tool.
Upgrade capabilities to take advantage of modern terminal emulation, so users get the advantage of colors, graphics, and interaction.
Understand and implement the conventions for well-behaved tools in your space, users can get to a sense of mastery quicker and avoid unpleasant surprises.

As I mentioned at the top, I’m working on a product called Relay that hopefully embodies these principles. Check out the introductory blog post to learn more and follow along as we develop our open-source CLI at github.com/puppetlabs/relay.

3 Commandments for CLI Design

🌈 eric sorenson 🎹🔈🎚 — Wed, 08 Apr 2020 00:00:00 +0000

photo: Wikimedia Commons (public domain)

In the beginning…

However, the old-school Unix utilities are not without their problems. The passage of time and inevitable feature creep have led to situations like this:

usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The humble ls command now has more options than there are letters in the alphabet! Plus, despite the proliferation of features, many advanced and useful options are off by default to preserve backwards compatibility. Other core utilities are in similar situations: find and tar are stuck with arcane, user-hostile syntaxes, and the man pages for modern distributions are littered with comments like this one from the BSD-derived macOS 10.15 grep manual: “This implementation supports those options; however, their use is strongly discouraged.” (Discouraged? By whom? Why?)

Fast Forward

Fortunately, though much remains constant in the world of command-line tools, much has changed. There’s a buzz of activity in the open-source world around re-examining the core user experience of CLIs with fresh eyes. Not only are the tiny, single-purpose tools getting a refresh, but a new generation of CLIs are emerging that provide entry points into complex cloud services and system tools. The best of these enable power users while retaining the user interaction paradigms that make the small tools great.

Enable Progressive Discovery — users are trying to solve a problem, but might start out knowing very little about the tool. The tool should guide them to a solution in iterative steps and provide plain-language help along the way. Users resorting to Google or StackOverflow is an anti-pattern here.
Go Upscale, Intelligently — the best tools infer the context they’re running in take advantage of modern capabilities if they can. For example, colorized text can greatly add to usability and aesthetics, but if the tool is being run in a pipeline, colors should be off to avoid polluting the output stream.
Be a Good CLI Citizen — there are a number of hard-fought CLI best practices that good tools follow, both to match users’ expectations and to peacefully co-exist with the rest of the ecosystem. It’s important for tool authors to be aware of these conventions, comply with as many as possible, and be deliberate about any significant divergence.

In the rest of the post I’ll provide some examples of each of these principles using CLI tools I’ve fallen in love with.

Enable Progressive Discovery

First, installing it is a breeze because the website has helpful instructions for all the major OSes with a one-click copy-to-clipboard command to get you started.

Go Upscale, Intelligently

My former colleague from Danger and all-around good human C J Silverio tweeted recently:

The rustlang ecosystem scene that’s making me happy right now: the retakes on ancient unix cli tools, with new approaches to info display.

bat

dua

exa

— Ceej is sheltering under cats (@ceejbot) February 22, 2020

# if rust stuff is found, use it
RBIN=$HOME/.cargo/bin
if [[-d $RBIN]]; then
  [[-f $RBIN/bat]] && for f in less more cat ; do
     alias $f=bat ;
  done
  [[-f $RBIN/dua]] && alias du=dua
  [[-f $RBIN/rg]] && alias grep=rg
  [[-f $RBIN/exa]] && alias ls=exa
  [[-f $RBIN/fd]] && alias find=fd
fi

Be a Good CLI Citizen

It’s tough to focus on a single tool as an example. Some of these conventions are defined in Jeff Dickey’s great post “12 Factor CLI Apps” about the Heroku CLI design philosophy, others from the GNU coreutils doc “Opening the software tool box”, and some are arguably matters of taste. Please comment if you think I’ve gotten something wrong or missed an important one!

Expect to operate as part of a pipeline and adjust behavior accordingly. Like the bat example above, turn off any fancy formatting if output isn’t a terminal so downstream tools see plain-text output for easy processing.
Don’t cross the streams! Closely related to pipeline operation, output should go to stdout and errors to stderr, so errors won’t be swallowed by redirection or intermingled with actual output.
Provide tabular and structured output formats for lists of data. For extracting only parts of a record, make it easy for users to split on whitespace to get just single fields. And no modern tool that outputs a lot of records in a table ought to be without a json output option.
For sophisticated CLIs, such as those which interact with remote services like Pulumi, think deeply about your command’s information architecture, then stick with it. A complex command invocation is like a grammatical sentence: it’s usually got a subject, a verb, and an object. For nontrivial commands there’s a hierarchy of these nouns and verbs that users put together to solve a problem. Either use a verb-object construction like kubectl get pods , where get is a top-level verb and the targets vary, or subject-verb construction like gcloud container clusters list (“clusters” is a subcommand of “container” and “list” is the verb).
Observe control conventions. Users frequently get things wrong, and sometimes that happens after they’ve hit the enter key on a command. The convention is that ^C (control-C) interrupts execution and exits the program, cleanly if possible. Similarly, ^Z (control-Z) suspends a program so that it can be paused or put in the background. Good CLI commands should honor these signals and behave predictably when they come in.
Don’t litter the filesystem. Use package managers and follow their conventions for the target operating system wherever possible. Similarly, follow conventions like XDG-Spec for guidance on where to install configuration files, docs, and supporting libraries. This kind of consistent packaging may take some additional work up front but it pays off in the long run when your software can be installed, upgraded, and config-managed alongside the rest of the user’s toolchain.

END OF FILE

When you’re done inputting a file into cat or its equivalents, a ^D (control-D) character indicates to the terminal that the end of the file’s been reached. Similarly, when ending a blog post it’s customary to summarize the points you’ve made and provide a stirring call to action for the reader. So, to recap the three top level design principles for modern command line interaction design:

Enable progressive discovery to lead users down a low-friction path to solving their problem with your tool.
Upgrade capabilities to take advantage of modern terminal emulation, so users get the advantage of colors, graphics, and interaction.
Understand and implement the conventions for well-behaved tools in your space, users can get to a sense of mastery quicker and avoid unpleasant surprises.

Thanks to Lee Briggs.

What’s Going on With Tekton? (Part 1)

🌈 eric sorenson 🎹🔈🎚 — Wed, 25 Mar 2020 00:00:00 +0000

Introduction

Tekton is a powerful tool for building continuous delivery pipelines using modern infrastructure. The project has been moving very quickly and has added a lot of awesome features recently, but due to the (necessary and wise) cancellation of conferences through the spring and summer of 2020 they have not gotten the attention they deserve. We’ve been working with Tekton as an upstream component of our devops automation service relay.sh, so I thought our new blog be a great chance to get the word out.

This 2-part blog series aims to give an overview of some core concepts for background and then dive into some more advanced technical topics and updates for Tekton. If you’re brand new to Tekton, check out Christie Wilson and Kim Lewandowski’s talk on “Next Generation CI/CD with GKE and Tekton” from Google Cloud Next 2019. If you want to skip ahead, you can go straight to Part 2.

Who’s Using Tekton?

It’s important to note that Tekton is more of a platform for building CD tools around, rather than a user facing product. The most advanced example of products built on Tekton is probably Jenkins X, a re-imagining of the ubiquitous Jenkins CI tool for cloud native applications. Jenkins X provides the user-facing GUI, a higher-level YAML dialect for describing pipelines, and a powerful GitOps workflow. To borrow git’s terminology, Jenkins X is the “porcelain”, while Tekton provides the “plumbing”: the pipeline execution engine that’s responsible for ordering the steps in a pipeline, launching pods that get the build/test/deploy work done, passing state and secrets around, etc. Ethan Jones at Cloudbees wrote a great blog post describing the history of the projects and the rationale behind the Jenkins X team’s decision to consolidate on Tekton as the one-and-only execution engine.

Similarly, here at Puppet, we’ve been using Tekton as the plumbing under the service formerly known as Project Nebula — now out of incubation and renamed Relay — for the last year. Relay lets you define workflows of operations tasks and trigger them by linking Relay to external services that you already use, like Github or PagerDuty. The goal is to fill in what we see as an emerging category of event-driven devops automation, now that desired-state and ad-hoc automation are well understood. When an event comes in to the service, Relay launches a container to figure out how to handle it (we’re calling this a trigger action). It can extract values from the incoming event and use them to fill in parameters in a workflow, which is launched as a Tekton pipeline on GCP. Similar to Jenkins X, workflows are written in simple YAML that the service translates into Tekton CRDs; Tekton then handles dependencies, ordering, and state management between workflow steps.

A Brief History of the (Tekton) Universe

Tekton is a project that evolved from an internal Google tool that used Knative to build and deploy software. In 2018, it was spun out as an independent project and donated to the Continuous Delivery Foundation, similar to Kubernetes’ donation to the CNCF.

The core component, Tekton Pipelines, runs as a controller in a Kubernetes cluster. It registers several custom resource definitions which represent the basic Tekton objects with the Kubernetes API server, so the cluster knows to delegate requests containing those objects to Tekton. These primitives are fundamental to the way Tekton works, and once you’ve got Tekton and the Tekton Dashboard up and running, you can view them like this:

> kubectl get crd | rg tekton | cut -f 1 -d ' '
clustertasks.tekton.dev
conditions.tekton.dev
extensions.dashboard.tekton.dev
pipelineresources.tekton.dev
pipelineruns.tekton.dev
pipelines.tekton.dev
taskruns.tekton.dev
tasks.tekton.dev

If the nomenclature here feels confusing, don’t feel bad — it is complicated! Each tool in the space uses slightly different terms; this is something we’re working on standardizing in the CDF Interoperability SIG. (We’d love your input!) Tekton’s usage of these terms — and the CRDs which implement the hierarchy of objects — maps closely to this list of CRDs (via the sig-interop Vocabulary definitions doc):

Step : a specific function to perform.
Task : is a collection of sequential steps you would want to run as part of your continuous integration flow. A task will run inside a pod on your cluster.
ClusterTask : Similar to Task, but with a cluster scope.
Pipeline : stateless, reusable, parameterized collection of tasks. Tasks are linked together in a Pipeline, which describes the end-to-end deployment for an application.
PipelineRun : an instantiation of a Pipeline definition, filling in the Pipeline’s parameters with concrete values
Pipeline Resource : objects that will be input to or output from tasks
Trigger : Triggers is a Kubernetes Custom Resource Definition (CRD) controller that allows you to extract information from event payloads (a “trigger”) to create Kubernetes resources.

Notable omissions from this list are “Steps”, which don’t have their own CRD because they’re the smallest unit of execution which are always contained inside a Task; plus Conditions and Dashboard Extensions, which are still optional and experimental — but very exciting! We’ll talk about these more in part 2.

If you’re working on an in-house CD platform, and especially if you’re looking at ways to deliver cloud-native applications with a toolchain that looks and feels like the apps it’s managing, Tekton could be a good fit. Read on if these capabilities sound intriguing…

What’s Going on With Tekton? (Part 2)

🌈 eric sorenson 🎹🔈🎚 — Wed, 25 Mar 2020 00:00:00 +0000

The Story So Far…

In Part 1 of this two-part blog, I talked about how we’re using Tekton Pipelines in building relay.sh, how Jenkins X is using it, and a brief history of the project and its terminology. In this part, I’ll dig into some of the cool advancements and features in the core Pipelines project and the broader ecosystem.

The ‘tkn‘ CLI tool

There’s been a ton of work lately going into the tkn command-line interface. I wanted to introduce it first, not just because there’ve been a number of very cool improvements in it recently, but also because we at Puppet are heavily invested in awesome command-line user experiences for our users, and tkn exhibits some great UX patterns that are worth looking at, even if you’re not using it directly.

You can install tkn via Homebrew or tarball; check out the README in the tektoncd/cli repo for the details. It uses the kubernetes client API, so if kubectl get pods --namespace=tekton-pipelines works for you, so will tkn. The first cool thing about the CLI that it supports both bash and zsh command completion, so typing tkn <Tab> will show you the list of subcommands it supports:

> tkn <Tab>
 -- command --
clustertask -- Manage clustertasks
completion -- Prints shell completion scripts
condition -- Manage conditions
eventlistener -- Manage eventlisteners
help -- Help about any command
pipeline -- Manage pipelines
pipelinerun -- Manage pipelineruns
resource -- Manage pipeline resources
task -- Manage tasks
taskrun -- Manage taskruns
triggerbinding -- Manage triggerbindings
triggertemplate -- Manage triggertemplates
version -- Prints version information

Many of these subcommands map directly back to the terminology and underlying CRDs I mentioned in the opening. It’s important to note that, by design, the tkn CLI is primarily a read-only interface, not a read-write one, so creation and editing of the objects still needs to go through kubectl. That’s because the creation of Tekton objects boils down to feeding YAML descriptions into the Kubernetes cluster API; modulo introducing a higher-order language like Relay or Jenkins X, there’s not much tkn can add over raw kubectl apply -f mypipeline.yaml. There are, however, several commands for controlling existing objects that are substantially nicer to use than their kubectl equivalents. For example, tkn pipeline start will kick off a new run of an existing pipeline, without you needing to craft a new YAML file for each one.

In addition to execution, the detailed output you get from tkn is vastly improved over thekubectl describe equivalents. For example, after running an example pipeline using conditions (more on those in a moment), I can get a compact list of recent runs and see useful information about the one I’m most interested in:

> tkn taskrun list
NAME STARTED DURATION STATUS
condtional-pr-then-check-gtp96 27 minutes ago 7 seconds Succeeded
condtional-pr-then-check-gtp96-file-exists-v9m2t 27 minutes ago 14 seconds Succeeded
condtional-pr-first-create-file-4n9zh 27 minutes ago 22 seconds Succeeded
steps-run-in-order-p6v7q 5 hours ago 9 seconds Succeeded
echo-hello-world-task-run 1 day ago 9 seconds Succeeded

> tkn taskrun describe condtional-pr-first-create-file-4n9zh [0/3547]
Name: condtional-pr-first-create-file-4n9zh
Namespace: default
Task Ref: create-readme-file
Service Account: default

🌡️ Status

STARTED DURATION STATUS
4 hours ago 22 seconds Succeeded

[...]

🦶 Steps

NAME STATUS
 ∙ write-new-stuff Completed
 ∙ create-dir-workspace-868lz Completed
 ∙ source-copy-pipeline-git-bpt22 Completed
 ∙ source-mkdir-pipeline-git-4lg5b Completed

🚗 Sidecars

No sidecars

(Yes, the emoji are part of the output! 😺)

The equivalent kubectl describe commands would spew out some verbose, unhelpful output and require lots of awk-ward manipulation in order to see the same level of usable info we see here.

Pipeline Improvements: Conditions

I mentioned above that the sample pipeline run we’re looking at makes use of Conditions. Conditions landed recently and are a welcome addition to Pipeline capabilities. Conditions extend Tekton’s core concept — launching task-specific containers and managing their success or failure — to enable decision-making in the middle of a pipeline run. Normally, a container’s entrypoint exiting with a non-zero exit code indicates that something went horribly wrong — it couldn’t clone a git repository, or unit tests failed, perhaps — but a container flagged as a condition failing just means that other Tasks marked as dependent on that container’s success will not be executed.

The pipeline run that generated the output above contains a condition that shows the utility and potential power of the feature.

apiVersion: tekton.dev/v1alpha1
kind: Condition
metadata:
  name: file-exists
spec:
  params:
    - name: 'path'
  resources:
    - name: workspace
      type: git
  check:
    image: alpine
    script: 'test -f $(resources.workspace.path)/$(params.path)'

This Condition uses the alpine image to run a really simple inline script to make sure a given file exists.

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: conditional-pipeline
spec:
  resources:
    - name: source-repo
      type: git
  params:
    - name: 'path'
      default: 'README.md'
  tasks:
    - name: first-create-file
      taskRef:
        name: create-readme-file
      resources:
        outputs:
          - name: workspace
            resource: source-repo
    - name: then-check
      conditions:
        - conditionRef: 'file-exists'
          params:
            - name: 'path'
              value: '$(params.path)'
          resources:
            - name: workspace
              resource: source-repo
              from: [first-create-file]
      taskRef:
        name: echo-hello

The Pipeline then creates a README.md file on line 14, then uses file-exists condition (starting on line 22) to test whether the file was created successfully. If so, it executes the echo-hello task. Both of these individual tasks are defined as separate Task objects to enable reuse:

---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: create-readme-file
spec:
  resources:
    outputs:
      - name: workspace
        type: git
  steps:
    - name: write-new-stuff
      image: ubuntu
      script: 'touch $(resources.outputs.workspace.path)/README.md'
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: echo-hello
spec:
  steps:
    - name: echo
      image: ubuntu
      script: 'echo hello'

There’s an elegance to this design that, to me anyway, expresses one of the coolest parts of working with Kubernetes. It distills a problem down to a set of declarative resources: has this container run? If not, run it. Then builds upwards: Did it exit successfully? If so, proceed with the next task.

The specification is described in the Conditions doc and there are several more conditional examples in the tektoncd/pipeline repo. Make sure you’re running the latest version as some of the implementation details are still solidifying as Tekton moves towards its beta release in early 2020.

Dashboard and Trigger Deliciousness

One of the best things about the Tekton community is the weekly working group calls. They’re open to anyone who joins the tekton-dev Google group (check out the details in the tektoncd/community repo), stay on-topic more than 95% of meetings I’ve seen, and present a really powerful and positive example of how multi-vendor open source ought to work. On any given Wednesday, there’ll be representatives from Google, VMware, IBM, Red Hat (well, a different part of IBM, anyway!), eBay, Apple, D2iQ, and many other vendor and end user organizations. While some of these companies’ agendas might diverge, the team overall works in a remarkably healthy and cooperative manner to get stuff done: from collaborative comments on design docs to encouraging messages on pull requests, from my perspective Tekton is a burgeoning open-source success story.

Contributor encouragement … and kittens!

On a recent working group call, Andrea Frittoli from IBM did a two-fer demo, showing not only how Tekton dogfoods Tekton for its own build and deployment pipeline, but how far the Tekton Dashboard has come in displaying the inner workings of the infrastructure.

Dashboard is an graphical add-on to Tekton, primarily developed by IBM folks as an upstream component of the Kabanero.io project, which (as you might expect) provides a graphical view into what’s going on across your Tekton deployment. I’ll give a brief rundown of Dashboard’s capabilities here; for an in-depth look, check out Adam Roberts’ blog post on the IBM developer blog “Why Now’s a Great Time to Use the Tekton Dashboard” (for whatever value of “now” happens to be true for you!)

As you can see from the sidebar, the Dashboard has kept pace with the new CRDs that the platform implements. One that’s particularly interesting is the highlighted EventListeners resources — the ones in the dogfooding repo are “raw” resources, but there’s an extension to the Dashboard that allows you to easily configure webhook events from external services so you can (for example) send Tekton an event when a Github PR is merged.

The Dashboard’s webhooks extension provides a friendly interaction layer into the Triggers system, but even without it the Tekton Triggers project is very cool. It allows Tekton to become responsive to events that originate outside of your Kubernetes cluster. This makes it much easier to integrate Tekton into your broader infrastructure, instead of requiring explicit requests against the k8s cluster API to trigger task and pipeline runs. The EventListeners , as in the screenshot above, provide an endpoint for external systems to talk to. As events come in, TriggerBindings transform fields form them into parameters forTriggerTemplates. The templates, in turn, generate resources like the Task and Pipeline runs that we’ve already seen. This level of dynamic instantiation truly levels Tekton’s capabilities up to the point where it can handle modeling the complexity of modern deployment scenarios.

The End

As you can tell by the length of this post, I’m really excited about all of the advancement in Tekton. As part of Relay, we’ve been trying to both keep abreast of the new features and contribute back wherever we can (in fact the cute kitten pic was from one of our pull requests!). The tkn CLI is an awesome example of using a familiar shell-based paradigm to interact with an API-driven service, Conditions bring a much-needed level of expressiveness to pipeline definitions, and we believe deeply in the future of event-driven automation.

Here are some links to follow up with the Tekton community and the related projects I mentioned in this series:

The tektoncd/community repo has links to our Slack workspace and mailing lists.
The Jenkins X community page has office hours and contact info.
The Kabanero project is a spicy way to get teams deploying Kubernetes apps faster.
We’d love to get your input on which services and tools you’d like to connect with Relay.