DEV Community: Cedric Clyburn

How to create cloud-native CI/CD Pipelines with Tekton

Cedric Clyburn — Thu, 04 Aug 2022 15:23:00 +0000

For modern applications that require CI/CD (continuous integration and continuous delivery/continuous deployment), how do you continuously deliver your code changes from source to production on Kubernetes? In this tutorial, you’ll learn how to use Tekton to clone your repository, build an image, and push it to the Docker Hub. Let's get started with Tekton, an open-source cloud native CI/CD solution, by creating a real-world CI/CD pipeline.

Introduction

The CI/CD process is fundamental to combining all parts of your code to validate, test, and build before releasing to production or any other stage. Tekton, originally started from the Knative project by Google developers, brings highly reusable and declarative CI/CD components to your cloud-native environment through Kubernetes CRDs. This can help you to concentrate more on developing without worrying about the complexities of the underlying implementation details.

Since Tekton is optimized for simplicity and reusability, let's look at the different building blocks that can help us build a continuous delivery pipeline.

-Step: The most basic component of Tekton, a Step is a Kubernetes container image that executes on a specific tool or input (ex. Writing or reading a file)

-Task: A sequence of steps to perform a single functionality that can be reused (ex. Cloning a repository)

-Pipeline: Series of tasks executed in series or parallel (ex. Clone a repository, build an image, and push it to a container registry)

Because these components are configured as Custom Resource Definitions on Kubernetes, you can create highly reusable pipelines, and Tekton is also compatible with Jenkins, Skaffold, Knative, and other CI/CD tools. Let’s take a look at getting started with Tekton, and start working on building our pipeline.

Getting started with Tekton

To start, let's install Tekton and any extras. What we’ll be using Tekton for is the classic container build process, where the Pipeline typically looks like the graphic below. Of course, there are always additional steps like testing, packaging, and deploying to our cluster, but for simplicity, our Pipeline will clone, build an image, and push an image to a container registry like the Docker Hub or Quay.io.

We'll work with Tasks from the Tekton Hub, create the configurations for our Docker Hub and ServiceAccount, and use a PipelineRun to instantiate our final Pipeline. Let’s begin!

Installing Tekton

Simply use a single kubectl command to install the latest version of Tekton Pipelines.

$ kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

You’ll also want to install the tkn CLI to interact with Tekton; instructions can be found in the documentation.

Additionally, feel free to install the Tekton dashboard to visualize our pipeline in a GUI using the Tekton Dashboard.

$ kubectl apply --filename https://storage.googleapis.com/tekton-releases/dashboard/latest/tekton-dashboard-release.yaml

The Tekton Dashboard is not exposed outside the cluster by default, but you can port-forward it using the following command and access it at localhost:9097.

$ kubectl port-forward -n tekton-pipelines service/tekton-dashboard 9097:9097

Creating Tekton Tasks

The tasks we’ll need to create our specific pipeline are already available at the Tekton Hub, a central portal for discovering and sharing Tekton resources, tasks, and pipelines. We’ll use the git-clone and buildah Tasks to clone our repository, build an image, and push the image to a container registry. Feel free to check out the details of these Tasks and their source code.

First, let’s install these Tasks with tkn hub install.

$ tkn hub install task git-clone && tkn hub install task buildah

Task git-clone(0.7) installed in default namespace
Task buildah(0.4) installed in default namespace

Now, let’s confirm the tasks have been successfully installed with tkn task list.

$ tkn task list

NAME                 DESCRIPTION              AGE
git-clone            These Tasks are Git...   2 minutes ago
buildah              Buildah task builds...   2 minutes ago

Image Push Secrets

To push the image to the Docker Hub, you’ll need to provide your credentials, which we can do by creating a file named secret.yaml, with the following content. Be sure to replace username and password with your Docker Hub credentials.

apiVersion: v1
kind: Secret
metadata:
  name: docker-secret
  annotations:
    tekton.dev/docker-0: https://index.docker.io/
type: kubernetes.io/basic-auth
stringData:
    username: username
    password: password
--- 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: docker-login
secrets:
  - name: docker-secret

This will create a Secret and ServiceAccount, which can now be used when running Tekton tasks, and we can apply it with kubectl.

$ kubectl apply -f secret.yaml

secret/docker-secret created

serviceaccount/docker-login created

Creating a Pipeline

Great! You’re ready to create a pipeline using the two git-clone and buildah tasks. Create a file named pipeline.yaml with the following content.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: example-pipeline
spec:
  params:
    - name: REPO
    - name: IMAGE
    - name: TAG
  workspaces:
    - name: workspace
  tasks:
    - name: fetch-repository
      taskRef:
        name: git-clone
      workspaces:
        - name: output
          workspace: workspace
      params:
        - name: url
          value: $(params.REPO)
        - name: deleteExisting
          value: "true"
    - name: build-push-image
      taskRef:
        name: buildah
      runAfter:
        - fetch-repository
      workspaces:
        - name: source
          workspace: workspace
      params:
        - name: IMAGE
          value: "$(params.IMAGE):$(params.TAG)"
        - name: FORMAT
          value: "docker"

This pipeline includes the two tasks we recently installed. First, a git-clone Task to clone our repository, using a url we’ll specify in a PipelineRun. Next, we build and push an image using the buildah Task, using an image and tag we’ll also specify in the PipelineRun to instantiate our Pipeline.

All we need to do to apply this Pipeline configuration to our cluster is use a standard kubectl command.

$ kubectl apply -f pipeline.yaml

pipeline.tekton.dev/example-pipeline created

Creating a PipelineRun

To run the pipeline, we can create a PipelineRun and specify our specific values for the repository, image, and reference to the pipeline. Create a file named pipelinerun.yaml with the following content. Being that Tekton Pipelines are highly reusable, you’re able to use any repository you’d like, and be sure to change your Docker Hub username.

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: tutorial-pipeline-run
spec:
  serviceAccountName: docker-login
  pipelineRef:
    name: example-pipeline
  params:
    - name: REPO
      value: "https://github.com/cedricclyburn/tekton-tutorial"
    - name: IMAGE
      value: "docker.io/cedricclyburn/tekton-tutorial"
    - name: TAG
      value: latest
  workspaces:
    - name: workspace
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

Now, let’s apply the file, and start the deployment with a kubectl command.

$ kubectl apply -f pipelinerun.yaml

pipelinerun.tekton.dev/tutorial-pipeline-run created

We’ve successfully run a pipeline, and to see the logs of the Pipeline, feel free to run the following command.

$ tkn pipeline logs -f

[1/3] STEP 1/5: FROM node:16-alpine AS deps
Resolved "node" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/node:16-alpine...

Checking the Results

Fantastic! We’ve successfully used a PipelineRun to run our Pipeline, so it’s time to check that everything worked properly. First, let’s use the Tekton Dashboard to view the logs and ensure the PipelineRun has succeeded. From your dashboard, navigate to the _PipelineRuns _tab to view your PipelineRuns, and select the tutorial-pipeline-run we’ve created.

From here, you can visualize the various Tasks of your Pipeline, including the parameters, results, logs, and status of each Task.

Finally, as everything we’ve seen looks good from the Pipeline logs using tkn pipeline logs and from the Tekton Dashboard, let’s run the image using Docker.

Feel free to use Podman, a daemonless and rootless container engine, instead as well, by replacing docker with podman in the following command.

$ docker run -p 3000:3000 docker.io/cedricclyburn/tekton-tutorial

When visiting localhost:3000, you’ll see the Next.js example application running like so. Congratulations!

Conclusion

Through this guide, you’ve learned how to work with Tekton to create cloud-native CI/CD Pipelines. You’ve also been able to build your own Pipeline to clone a repository, build an image, and push that image to the Docker Hub, all by using Tasks from the Tekton Hub. There are plenty more reusable Tasks and Pipelines there, but I also encourage you to learn about Tekton Triggers (automatically running a Pipeline), other projects of the Continuous Delivery Foundation and the Tekton documentation for more information!

// Detect dark theme var iframe = document.getElementById('tweet-1549061815165825024-758'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1549061815165825024&theme=dark" }

Feel free to follow me on Twitter @cedricclyburn for more cloud-native tutorials!

Containers without Docker (podman, buildah, and skopeo)

Cedric Clyburn — Tue, 19 Jul 2022 20:07:00 +0000

While Docker is commonly the standard when it comes to containers, we're starting to see why alternatives like Podman are quickly becoming popular replacements for container-based development. Although Docker has been around for almost 10 years and standardized containerization, Podman has some huge advantages, namely being "daemonless" (not needing systemd or any other service to run in the background). Are there alternatives to Docker? Yes, and let's take a look at how we can get started with Podman, Buildah, and Skopeo.

Introduction

Docker has been an innovator in how developers build and deploy applications, and is a good tool. From building images, pushing/pulling images from registries, and running images, Docker does it all. However, Docker is a huge monolithic application and relies on a heavy daemon, which, if at a point failed, all child processes would become unreachable.

This is what Podman, an open-source daemonless and rootless container engine, was developed with in mind. Podman runs using the runC container runtime process, directly on the Linux kernel, and launches containers and pods as child processes. In addition, it was developed for the Docker developer, with most commands and syntax seamlessly mirroring Docker's. Buildah, an image builder, and Skopeo, the image utility tool, are both complimentary to Podman as well, and extend the range of operations able to be performed.

Due to Docker being such a large tool, we can break it down into a few components, mainly, this includes the container engine, image builder, and image distribution. Instead of relying on Docker, let’s take a look at some light-weight replacements that can achieve the same functionality.

Podman

Podman is a daemonless and rootless container engine, allowing you to run, manage, and interact with containers. The commands are the same as Docker due to the standards of the Open Container Initiative (OCI), and you can even alias Docker to Podman (alias docker=podman). While the Docker daemon normally runs as root, a long-standing security concern, Podman can be run in rootless mode. Podman even includes the functionality to orchestrate containers with Kubernetes!

Getting started with Podman

Firstly, let’s install Podman. Based on your runtime, you can use the various installation methods in the documentation. You can even easily use Podman on Mac using brew install podman and Windows using this blog.



$ sudo dnf -y install podman

Run a httpd container



$ podman run -dt -p 8080:80/tcp docker.io/library/httpd

Trying to pull docker.io/library/httpd:latest...
Getting image source signatures
Copying blob d6bc17b4451a done 
…

Check container status



$ podman ps

CONTAINER ID  IMAGE                           COMMAND           CREATED         STATUS             PORTS                 NAMES

5d5f92a59ea3  docker.io/library/httpd:latest  httpd-foreground  24 seconds ago  Up 25 seconds ago  0.0.0.0:8080->80/tcp  happy_beaver

Access your application! We could simply do a curl command, or access the IP of our application with port 8080 on a browser, and we’ll see our httpd server running.



$ curl localhost:8080

<html><body><h1>It works!</h1></body></html>

Buildah

Buildah is a daemonless and rootless image builder tool complimentary to Podman that produces OCI compliant images. It’s able to build images from Dockerfile, and can be run directly inside of containers.

Getting started with Buildah

Let’s install Buildah, which can be done by checking the installation documentation. Just like Podman, Buildah is shipped on Fedora 35/36, RHEL 8+, CentOS, OpenSUSE, and Ubuntu, and more.



$ sudo dnf -y install buildah

Build an image from Dockerfile. While the default command to build any container image is buildah bud -t tag-name . in a directory with a Dockerfile, let’s say we’re working on a Next.js application. Firstly, let’s download the bootstrap using npx.



$ npx create-next-app --example with-docker nextjs-docker

$ cd next-js docker

Now, we’ve got a Next.js application, with a Dockerfile inside ready to build our app. Let’s go ahead and build our container image.



$ buildah build -t nextjs-docker .

[1/3] STEP 1/5: FROM node:16-alpine AS deps

Resolved "node" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)

Trying to pull docker.io/library/node:16-alpine…

Successfully tagged localhost/nextjs-docker:latest

Check image status.



$ buildah images

REPOSITORY               TAG         IMAGE ID      CREATED         SIZE

localhost/nextjs-docker  latest      162bee38beb9  8 seconds ago   118 MB

Run container and check the installation, looks great!



$ podman run -p 3000:3000 nextjs-docker

Skopeo

Skopeo is an image utility tool that complements Podman and Buildah, allowing you to remotely inspect images, copy images between registries, and more. Like the others, it doesn’t require a daemon to be running or root privileges, and can work with OCI compatible images.

Getting started with Skopeo

Let’s install Skopeo, which can be done using the installation documentation.



$ sudo dnf -y install skopeo

Push our image to a remote registry. If we haven’t already done so, let’s upload the image we just built with Buildah to the Docker hub.



$ podman login docker.io

$ podman push nextjs-docker docker.io/cedricclyburn/nextjs-docker

Copy image to a different registry. Recently, Docker Hub rate limits and paid tier changes have encouraged engineers to start using alternative image registries like Quay.io. Let’s take our existing image from Docker, and copy it over to Quay.io.



$ skopeo login quay.io

$ skopeo copy docker://cedricclyburn/nextjs-docker:latest docker://quay.io/cedric-clyburn/nextjs-docker:latest

Inspect the image. Skopeo allows us to inspect properties or configuration of an image on remote repositories using skopeo inspect.



$ skopeo inspect docker://quay.io/cedric-clyburn/nextjs-docker

{

    "Name": "quay.io/cedric-clyburn/nextjs-docker",

    "Digest": "sha256:779bf91bd2d407b4db9e7e7035cc77dfbd0f2cbd435067a40f485960d2889ded",

    "RepoTags": [

        "latest"

    ],

    "Created": "2022-07-19T18:30:29.872420186Z",

…

Conclusion

As we’ve seen, you can use Podman, Buildah, and Skopeo as replacements to the traditional Docker workflow, without the use of a daemon or root privileges. There are plenty of great advantages that result from using these tools, and due to the increase of developer adoption there’s only more to come. I believe there’s a future for containers without Docker, and Podman (as well as the Buildah and Skopeo family) is a great alternative to work with.

// Detect dark theme var iframe = document.getElementById('tweet-1329559076058132483-843'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1329559076058132483&theme=dark" }

Note: This content comes from one of my DevNation Tech Talks, and the full presentation + slides can be found here!