DEV Community: Ken Moini

Up and running with Patternfly 4

Ken Moini — Sun, 14 Jun 2020 22:26:53 +0000

If you've used any Red Hat technologies in the last few years, you may have noticed a common theme. This standardized design framework is called Patternfly and is based on the Bootstrap framework. Patternfly is one of the over 1 million Open-Source projects that are backed by Red Hat.

I've used Patternfly 3 before, which was basically a Bootstrap 3 theme. Migrating to Patternfly 4 had a few changes that broke my previous workflow and assets. It's nothing really difficult to address, though the documentation is sometimes difficult to navigate at first - it makes sense once you start to use Patternfly and understand how it was redesigned.

The Cliff Notes

Patternfly 4 now has two variants: a core HTML/CSS only version, and one based on React. I have been working on updating a PF3 site to PF4, so I used the core HTML/CSS only version - also because I don't care for Facebook and their technologies.
Note that the core HTML/CSS version does NOT come with the Javascript needed to make all those assets and components work! Accordions, dropdowns, modals - they don't work! PF4 is BYOJS or Bring-Your-Own-JavaScript. This was confusing at first while I wondered where the hell the JS files were...
Development instructions tend to be unclear, with the "missing" JS this spent many cycles.

Thankfully for you, I've got the cheat codes and notes so you don't have to spend as many cycles getting started with Patternfly 4!

Building with Patternfly 4

In this guide we'll be going over how to use the core HTML/CSS version of Patternfly 4 in a new web project and how to bring in the needed JavaScript and glue to make it all work!

1) Create new environment

First off, let's start with a blank canvas - assuming you already have Git, NodeJS, and NPM installed:

mkdir my-awesome-site && cd my-awesome-site
git init
npm init

After that, feel free to modify the package.json file that is created to match your new project specfications. I'd also suggest making your first commit after the init as well!

2) Pull in needed packages

Now that we have a new environment, let's bring in the Patternfly 4 framework and a few other packages that'll make it all work nicely.

npm i @patternfly/patternfly --save
npm i bootstrap --save
npm i jquery --save
npm i jquery-slimscroll --save #optional

That'll pull in the Patternfly 4 framework, Bootstrap and its dependencies such as jQuery, and there's an optional SlimScroll package to make things look a bit nicer.

3) Using the Patternfly 4 Styles

So at this point we have the assets needed to include Patternfly 4 into our project. All we need to do is include the CSS files from the node_modules/@patternfly/patternfly/ directory:

<link rel="stylesheet" href="/node_modules/@patternfly/patternfly/patternfly.min.css" />
<link rel="stylesheet" href="/node_modules/@patternfly/patternfly/patternfly-addons.css" />

You'll also see the SCSS files available - you can include the Patternfly package in your own SCSS file, override your desired styles, and compile the CSS assets as part of your own build process if you already have something like Webpack already set up with an SCSS compiler.

Alternatively, you can also modify the Patternfly package and build a new custom packaged version of Patternfly by running npm run build in the Patternfly package directory - note that this is generally an anti-pattern, you don't really want to modify the package as much as you'd want to extend it because your changes could easily be overwritten next time you run npm to manage packages.

4) Adding the JavaScript Glue

Patternfly is a design framework - the functionality of the designed components isn't defined (which is weird from a UX perspective...). This basically means that the controlling functions aren't defined and we need to bring in our own. Thankfully, Patternfly is built on Bootstrap so many of the same JavaScript bits will work with little modification.

Next, we'll include the needed JavaScript files from jQuery and Bootstrap - note that we're not including the Bootstrap {S}CSS, just the JavaScript.

<!-- jQuery -->
<script type="text/javascript" src="/node_modules/jquery/dist/jquery.min.js"></script>

<!-- Bootstrap -->
<script type="text/javascript" src="/node_modules/bootstrap/dist/js/bootstrap.bundle.min.js"></script>

<!-- SlimScroll -->
<script type="text/javascript" src="/node_modules/jquery-slimscroll/jquery.slimscroll.min.js"></script>

5) Real-world Example

Now that we have all our needed components, let's go ahead and build a couple pages with Patternfly 4.

The demos on the main public Patternfly 4 site have the source minified so it's difficult to build off those examples. However, the build available at https://pf4.patternfly.org/ is not minified and perfect for glancing from the source!

Page with Side Nav and Cards

<!DOCTYPE html>
  <html lang="en-us">
    <head>
      <meta charSet="utf-8"/>
      <meta http-equiv="x-ua-compatible" content="ie=edge"/>
      <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"/>
      <title data-react-helmet="true">HTML - Page default nav</title>
      <link rel="stylesheet" href="/node_modules/@patternfly/patternfly/patternfly.min.css" />
      <link rel="stylesheet" href="/node_modules/@patternfly/patternfly/patternfly-addons.css" />
      <style rel="stylesheet" type="text/css">
        #mainSideNav.collapse:not(.show) {
          display: none;
        }
        #mainSideNav.collapse.show {
          display:grid;
          overflow: hidden;
        }
        #mainSideNav.collapsing{
          height: 100vh !important;
        }
        .pf-c-dropdown__menu {
          display:none;
        }
        .pf-c-dropdown__menu.show {
          display:inherit;
        }
      </style>
    </head>
    <body>
      <div id="___gatsby">
        <div style="outline:none" tabindex="-1" id="gatsby-focus-wrapper">
          <main class="ws-site-root">
          <div class="ws-fullscreen-example">
            <div class="pf-c-page" id="page-default-nav-example">
              <a class="pf-c-skip-to-content pf-c-button pf-m-primary" href="#main-content-page-default-nav-example">Skip to content</a>
              <header class="pf-c-page__header">
                <div class="pf-c-page__header-brand">
                  <div class="pf-c-page__header-brand-toggle">
                    <button class="pf-c-button pf-m-plain" type="button" id="page-default-nav-example-nav-toggle" aria-label="Global navigation" aria-expanded="true" data-toggle="collapse" data-target="#mainSideNav" aria-controls="mainSideNav">
                      <i class="fas fa-bars" aria-hidden="true"></i>
                    </button>
                  </div>
                  <a class="pf-c-page__header-brand-link">
                    <img class="pf-c-brand" src="/assets/images/PF-Masthead-Logo.svg" alt="PatternFly logo" />
                  </a>
                </div>
                <div class="pf-c-page__header-tools">
                  <div class="pf-c-page__header-tools-group">
                    <div class="pf-c-page__header-tools-item pf-m-hidden pf-m-visible-on-lg">
                      <button class="pf-c-button pf-m-plain" type="button" aria-label="Settings">
                        <i class="fas fa-cog" aria-hidden="true"></i>
                      </button>
                    </div>
                    <div class="pf-c-page__header-tools-item pf-m-hidden pf-m-visible-on-lg">
                      <button class="pf-c-button pf-m-plain" type="button" aria-label="Help">
                        <i class="pf-icon pf-icon-help" aria-hidden="true"></i>
                      </button>
                    </div>
                  </div>
                  <div class="pf-c-page__header-tools-group">
                    <div class="pf-c-page__header-tools-item pf-m-hidden-on-lg">
                      <div class="pf-c-dropdown">
                        <button class="pf-c-dropdown__toggle pf-m-plain" type="button" id="page-default-nav-example-dropdown-kebab-right-aligned-1-button" aria-expanded="false" aria-label="Actions">
                          <i class="fas fa-ellipsis-v" aria-hidden="true"></i>
                        </button>
                        <ul class="pf-c-dropdown__menu pf-m-align-right" aria-labelledby="page-default-nav-example-dropdown-kebab-right-aligned-1-button" hidden>
                          <li>
                            <a class="pf-c-dropdown__menu-item" href="#">Link</a>
                          </li>
                          <li>
                            <button class="pf-c-dropdown__menu-item" type="button">Action</button>
                          </li>
                          <li>
                            <a class="pf-c-dropdown__menu-item pf-m-disabled" href="#" aria-disabled="true" tabindex="-1">Disabled link</a>
                          </li>
                          <li>
                            <button class="pf-c-dropdown__menu-item" type="button" disabled>Disabled action</button>
                          </li>
                          <li class="pf-c-divider" role="separator"></li>
                          <li>
                            <a class="pf-c-dropdown__menu-item" href="#">Separated link</a>
                          </li>
                        </ul>
                      </div>
                    </div>
                    <div class="pf-c-page__header-tools-item pf-m-hidden pf-m-visible-on-md">
                      <div class="pf-c-dropdown">
                        <button class="pf-c-dropdown__toggle pf-m-plain" type="button" id="page-default-nav-example-dropdown-button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
                          <span class="pf-c-dropdown__toggle-text">John Smith</span>
                          <span class="pf-c-dropdown__toggle-icon">
                            <i class="fas fa-caret-down" aria-hidden="true"></i>
                          </span>
                        </button>
                        <div class="pf-c-dropdown__menu" aria-labelledby="page-default-nav-example-dropdown-button">[Panel contents here]</div>
                      </div>
                    </div>
                  </div>
                  <img class="pf-c-avatar" src="/assets/images/img_avatar.svg" alt="Avatar image" />
                </div>
              </header>
              <div class="pf-c-page__sidebar collapse show" id="mainSideNav">
                <div class="pf-c-page__sidebar-body">
                  <nav class="pf-c-nav" id="page-default-nav-example-primary-nav" aria-label="Global">
                    <ul class="pf-c-nav__list">
                      <li class="pf-c-nav__item">
                        <a href="#" class="pf-c-nav__link">System panel</a>
                      </li>
                      <li class="pf-c-nav__item">
                        <a href="#" class="pf-c-nav__link pf-m-current" aria-current="page">Policy</a>
                      </li>
                      <li class="pf-c-nav__item">
                        <a href="#" class="pf-c-nav__link">Authentication</a>
                      </li>
                      <li class="pf-c-nav__item">
                        <a href="#" class="pf-c-nav__link">Network services</a>
                      </li>
                      <li class="pf-c-nav__item">
                        <a href="#" class="pf-c-nav__link">Server</a>
                      </li>
                    </ul>
                  </nav>
                </div>
              </div>
              <main class="pf-c-page__main" tabindex="-1" id="main-content-page-default-nav-example">
                <section class="pf-c-page__main-breadcrumb">
                  <nav class="pf-c-breadcrumb" aria-label="breadcrumb">
                    <ol class="pf-c-breadcrumb__list">
                      <li class="pf-c-breadcrumb__item">
                        <a href="#" class="pf-c-breadcrumb__link">Section home</a>
                      </li>
                      <li class="pf-c-breadcrumb__item">
                        <span class="pf-c-breadcrumb__item-divider">
                          <i class="fas fa-angle-right" aria-hidden="true"></i>
                        </span>
                        <a href="#" class="pf-c-breadcrumb__link">Section title</a>
                      </li>
                      <li class="pf-c-breadcrumb__item">
                        <span class="pf-c-breadcrumb__item-divider">
                          <i class="fas fa-angle-right" aria-hidden="true"></i>
                        </span>
                        <a href="#" class="pf-c-breadcrumb__link">Section title</a>
                      </li>
                      <li class="pf-c-breadcrumb__item">
                        <span class="pf-c-breadcrumb__item-divider">
                          <i class="fas fa-angle-right" aria-hidden="true"></i>
                        </span>
                        <a href="#" class="pf-c-breadcrumb__link pf-m-current" aria-current="page">Section landing</a>
                      </li>
                    </ol>
                  </nav>
                </section>
                <section class="pf-c-page__main-section pf-m-light">
                  <div class="pf-c-content">
                    <h1>Main title</h1>
                    <p>This is a demo of the Page component.</p>
                  </div>
                </section>
                <section class="pf-c-page__main-section">
                  <div class="pf-l-gallery pf-m-gutter">
                    <div class="pf-l-gallery__item">
                      <div class="pf-c-card">
                        <div class="pf-c-card__body">This is a card</div>
                      </div>
                    </div>
                    <div class="pf-l-gallery__item">
                      <div class="pf-c-card">
                        <div class="pf-c-card__body">This is a card</div>
                      </div>
                    </div>
                    <div class="pf-l-gallery__item">
                      <div class="pf-c-card">
                        <div class="pf-c-card__body">This is a card</div>
                      </div>
                    </div>
                  </div>
                </section>
              </main>
            </div>
          </div>
        </main>
      </div>
    </div>
    <!-- jQuery -->
    <script type="text/javascript" src="/node_modules/jquery/dist/jquery.min.js"></script>

    <!-- Bootstrap -->
    <script type="text/javascript" src="/node_modules/bootstrap/dist/js/bootstrap.bundle.min.js"></script>

    <!-- SlimScroll -->
    <script type="text/javascript" src="/node_modules/jquery-slimscroll/jquery.slimscroll.min.js"></script>

    <!-- Custom Javascript -->
    <script type="text/javascript">
    <!--
      jQuery(document).ready(function() {
        jQuery('.pf-c-page__sidebar-body').slimScroll({
          height: '100%',
          width: 'var(--pf-c-page__sidebar--Width)',
        });
      });
    //-->
    </script>
  </body>
</html>

That should get you something that looks like this:

Dropdown and Side Nav Toggle

There are a few things you should note in the example above:

The #page-default-nav-example-nav-toggle element has the needed data- and aria- attributes defined to control the display toggle of #mainSideNav
There is CSS defined to add some extra smooth-ness to the side navigation toggle.
The Joshua Smith dropdown has the needed data- and aria- attributes defined to control the display of the dropdown.
There is some CSS to control when the dropdown is displayed.
There is additional JavaScript to give the side navigation a nice SlimScroll.

Basically, the Bootstrap component actions can be directly implemented in Patternfly 4 with the use of the data- attributes, or with initialization in JavaScript.

Next Steps

From here, you can go about referencing the Patternfly 4 core examples available here: https://pf4.patternfly.org/

With the HTML set in your document, add the needed glue from the Bootstrap 4 documentation: https://getbootstrap.com/docs/4.5/components/alerts/

After you get comfortable with the CSS prefixes and how things are built and set up you can move onto building it as a native Gatsby site or including the SCSS in your own builds to extend the style guidelines in your own ways!

Custom Kubernetes Ingress Default Backend and Error Pages

Ken Moini — Mon, 25 May 2020 02:40:27 +0000

When you're setting up a Kubernetes cluster, one of the choices you need to make is what Ingress to use - or what combination of Ingresses (Ingressi?) to use.

I've used a number of Ingress Controllers in Kubernetes clusters on different clouds, and they all have their place and reason. The classic choice however, is the now native Ingress option provided by the Kubernetes project, the nginx-ingress controller.

Wait, what is an Ingress?

In the terms of a Kubernetes cluster, an Ingress Controller is a cluster resource that will route traffic into the services in the cluster, from external requests. This is how mycutedogs.com gets translated and routed to the side-sites/mycutedogs-com namespace/service in the cluster.

Blank and Bleak

When you deploy the Nginx Ingress Controller, you'll see some very basic error pages such as this:

But instead, what if you could make it look something like this:

How this all works

So when you deploy the Nginx Ingress Controller, it listens on the edge of your Kubernetes cluster for traffic. If it does not have a matching service to route to, it'll throw an error. Those errors are served traditionally from the nginx-ingress-controller container, but you can override where those errors are served from with some modifications to the Deployment and ConfigMap.

In order to set up your own custom error pages you'll need to fork my repo, build your own copy of the container image with your own error HTML files, and do some modifications to the corresponding Deployment Config.

Optional: Modify the error pages

If you'd like to set your own default-backend custom error pages, you'll need to do the following steps - if not and you'd just like to use the more spiffy versions I have available, continue to the Deployment steps

Fork my repo: https://github.com/kenmoini/custom-nginx-ingress-errors
Modify the error pages in the www/ directory as you see fit
Connect to Docker Hub, Quay.io, etc to create your own available image
Modify the k8s-deployment.yaml file in the repo to point to your own image.

Deployment

These instructions assume that you deployed the Ingress Controller in the default ingress-nginx namespace.

1) Clone down my repo: https://github.com/kenmoini/custom-nginx-ingress-errors



git clone https://github.com/kenmoini/custom-nginx-ingress-errors
cd custom-nginx-ingress-errors

2) Deploy to your Kubernetes cluster - the YAML file already targets the nginx-ingress namespace:



kubectl apply -f k8s-deployment.yaml

3) Modify the ingress-nginx/ingress-nginx-controller Deployment and set the value of the --default-backend-service flag to the namespace/svc-name of the newly created error backend, which should be ingress-nginx/nginx-errors by default.



export KUBE_EDITOR="nano" # optional, switch to the Nano editor
kubectl edit deployment/ingress-nginx-controller -n ingress-nginx

---
apiVersion: apps/v1
kind: Deployment
[...]
spec:
  [...]
  template:
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --default-backend-service=$(POD_NAMESPACE)/nginx-errors
[...]

4) Edit the ingress-nginx/nginx-configuration ConfigMap and add the key:value pair of "custom-http-errors": "404,500,503":



kubectl edit cm/nginx-configuration -n ingress-nginx

---
apiVersion: v1
kind: ConfigMap
metadata:
 [...]
data:
  custom-http-errors: "404,500,503"

5) Test your new error pages by navigating to the Ingress Load Balancer:



$ kc get services -n ingress-nginx

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE

ingress-nginx   LoadBalancer   10.245.31.157   173.28.135.40   80:32563/TCP,443:30965/TCP   18d

nginx-errors    ClusterIP      10.245.17.217   <none>           80/TCP                       117m

$ curl -sS http://173.28.135.40

<!DOCTYPE html>

<html lang="en">

<head>

<meta charset="utf-8">

<meta http-equiv="X-UA-Compatible" content="IE=edge">

<meta name="viewport" content="width=device-width, initial-scale=1">

<title>Error 404</title>

[...]

Bonus: Docker Hub Pre-Push Hook

This repo also has an extra file in hooks/pre-hook which creates another tag with the Git repo commit - it's pretty neat!

Jenkins Configuration as Code with Docker and Kubernetes

Ken Moini — Wed, 06 May 2020 21:08:01 +0000

So there are a few tools I use in my CI/CD tool-chain - Ansible, GitLab, and Jenkins. They all have their strengths, and weaknesses, it's all about using the right tool for the right job. Sure, you could use a flat-head driver on a Philips but, should you?

In setting these things up time and time, and time, and time, and...

...and time again, I've gotten good at doing it fast, and with as little manual intervention as possible. Most of these tools have some sort of API or CLI to interface with and automate but Jenkins provides a (now more mature) Configuration as Code (JCasC, CaC, etc) plugin. The JCasC plugin allows you to take configuration and store it as YAML, drop it into a directory (or mount into a ConfigMap/Volume in Kubernetes/OpenShift), and Jenkins will pick up that configuration and apply it, so long as the needed plugins are there.

Jenkins Plugins

So, plugins in Jenkins get a bad rep - as do WordPress plugins, and many other libraries, packages, and plugins created by Free Open-Source Software (FOSS) contributors. Sometimes a a plugin doesn't get updated, or a dependency has a vulnerability that is made worse by the parent plugin, or the developer moves on and it just stagnates for years. This is unfortunately the double-edged sword of FOSS, which is why it's often wise to go for supported Open-Source such as that from Red Hat and CloudBees. In fact, I'll be using CloudBees Jenkins Distribution which is a much more properly built Jenkins - fo free.

Anywho, I won't go into the deployment of Jenkins, there are a dozen different ways to do it and to do it properly is a rather large article in and of itself - if there's interest I'll write one though.

Let's assume that you as well have Jenkins, or CloudBees Jenkins Distribution, installed and accessible - great! After the Setup Wizard, one of the first things you'll want to do is start stuffing it with plugins - thankfully I have a handy script for you to do it automatically!

Grab that, drop it into your $JENKINS_HOME (or somewhere wx'able), and pass it a list of plugins you'd like to install, such as:



/var/jenkins_home/jenkins-plugin-stuffer.sh anchore-container-scanner ansible configuration-as-code docker-commons docker-java-api docker-plugin docker-workflow durable-task generic-webhook-trigger git git-client gitlab-plugin git-server kubernetes kubernetes-cli kubernetes-client-api kubernetes-credentials matrix-auth nexus-jenkins-plugin pipeline-aggregator-view pipeline-build-step pipeline-graph-analysis pipeline-input-step pipeline-milestone-step pipeline-model-api pipeline-model-declarative-agent pipeline-model-definition pipeline-model-extensions pipeline-rest-api pipeline-stage-step pipeline-stage-view pipeline-utility-steps role-strategy sonar workflow-aggregator workflow-api workflow-basic-steps workflow-cps workflow-cps-global-lib workflow-durable-task-step workflow-job workflow-multibranch workflow-scm-step

Whew - yeah. Anywho, that's a decent "cloud-native" plugin setup, run that and restart Jenkins, then on to the Configuration as Code ...configuration!

Set JCasC Path

Before the Configuration as Code plugin can load the configuration manifests, you need to tell Jenkins where to load it from! To do so, navigate to the Manage Jenkins > Configuration as Code section of the Web UI.

Add in a directory with where we're going to store the JCasC manifests - I like to do $JENKINS_HOME/jcasc

Click Apply, Save, and now we can start by adding our JCasC manifests!

JCasC for Docker

One of the things you may want to do with your Jenkins instance is build/interact with/run container images via Docker. We've got the plugins set up, now we just need to add the configuration manifest, say a file like $JENKINS_HOME/jcasc/cloud-docker.yml:



jenkins:
  clouds:
    - docker:
        name: "docker"
        dockerApi:
          dockerHost:
            uri: "unix:///var/run/docker.sock"
        templates:
          - labelString: "docker-agent"
            dockerTemplateBase:
              image: "jenkins/slave"
              volumes:
                - /var/run/docker.sock:/var/run/docker.sock
            remoteFs: "/home/jenkins/agent"
            connector:
              attach:
                user: "jenkins"
            instanceCapStr: "10"
            retentionStrategy:
              idleMinutes: 1

This is a great starter configuration if you're using the local Docker Unix Socket (default config when installing Docker on Linux/Mac). Save this file and when Jenkins is restarted the configuration will be loaded allowing you to run Pipelines with the Docker agents. Read more about that here: https://www.jenkins.io/doc/book/pipeline/docker/

JCasC for Kubernetes

Maybe instead of using Docker to build applications and run pipelines in Jenkins, you'd like to use Kubernetes and maybe even Kaniko? No problem, we can do this pretty easily now:



jenkins:
  clouds:
  - kubernetes:
      connectTimeout: 5
      containerCapStr: "10"
      credentialsId: "my-k8s-token"
      serverUrl: "https://k8s.api.example.com"
      skipTlsVerify: true
      jenkinsUrl: "https://jenkins.example.com/jenkins"
      maxRequestsPerHostStr: "32"
      name: "kubernetes"
      readTimeout: 15
      templates:
      - containers:
        - args: "^${computer.jnlpmac} ^${computer.name}"
          image: "registry.access.redhat.com/openshift3/jenkins-slave-maven-rhel7"
          livenessProbe:
            failureThreshold: 0
            initialDelaySeconds: 0
            periodSeconds: 0
            successThreshold: 0
            timeoutSeconds: 0
          name: "jnlp"
          workingDir: "/tmp"
        hostNetwork: false
        label: "maven"
        name: "maven"
        workspaceVolume:
          emptyDirWorkspaceVolume:
            memory: false
        yamlMergeStrategy: "override"
      - containers:
        - args: "^${computer.jnlpmac} ^${computer.name}"
          image: "registry.access.redhat.com/openshift3/jenkins-agent-nodejs-8-rhel7"
          livenessProbe:
            failureThreshold: 0
            initialDelaySeconds: 0
            periodSeconds: 0
            successThreshold: 0
            timeoutSeconds: 0
          name: "jnlp"
          workingDir: "/tmp"
        hostNetwork: false
        label: "nodejs"
        name: "nodejs"
        workspaceVolume:
          emptyDirWorkspaceVolume:
            memory: false
        yamlMergeStrategy: "override"

Save that file as $JENKINS_HOME/jcasc/cloud-kubernetes.yml, restart Jenkins, and enjoy the new ability to deploy pipelines on Kubernetes! Read more about the Jenkinsfile side of things here: https://plugins.jenkins.io/kubernetes/

Wrap

That's that! Within a short time you can deploy Jenkins or CloudBees Jenkins Distribution, add all your plugins with a few commands, set up configuration from code and get to building applications with Jenkins, Docker, and Kubernetes!

Deploy Harbor Container Registry - with Replication!

Ken Moini — Tue, 28 Apr 2020 23:42:30 +0000

So I have a rather robust DevOps platform at home in my lab - private GitLab, Kubernetes, registry, and so on. Now that I'm pushing to a public and production Kubernetes cluster in the cloud, I have a very interesting problem - getting my container images from my private lab network, which has an inaccessible subnet and custom TLD, pulled into the public Kubernetes clusters...

I cooooould create a VPN between my pfSense router and the Kubernetes cluster which would then allow routing to the private container registry. Problem with that is that I also use a custom Certificate Authority in my lab that works great, but since I use a managed Kubernetes service, I can't easily add my certificates to those nodes.

So I guess I have to deploy another Harbor Container Registry in the cloud to mirror my images publicly...

Deploy the VM

I'm working with DigitalOcean's managed Kubernetes service - with their Marketplace of apps and the GitLab integration I use, it's pretty easy to get up and going and use. That's key because I was getting tired of managing my own clusters and their constantly changing deployments.

I'm also going to deploy this Harbor Container Registry in DigitalOcean, in the same Region and Availability Zone so that I can use private networking between the Kubernetes cluster and the registry.

You can quickly deploy the same sorta VM I'm using by clicking the following link which will take you to the DigitalOcean Cloud Panel:

https://cloud.digitalocean.com/droplets/new?i=e07bd5&size=s-2vcpu-4gb&region=nyc1&distro=centos&distroImage=centos-7-x64&options=install_agent

(That's not a referral link, but this is: https://m.do.co/c/9058ed8261ee)

Configuring the VM

So to deploy Harbor, we're going to use Docker and Docker Compose. This is going to make things very, very easy...connect to that VM you just made and run:

sudo yum install epel-release -y
sudo yum update -y
reboot

Once we've done a quick system update and reboot to bring in any new kernels, we can continue with confidence. Speaking of confidence, let's set some security basics:

sudo yum install fail2ban firewalld wget nano -y
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo systemctl enable fail2ban
sudo systemctl enable firewalld
sudo systemctl start fail2ban
sudo systemctl start firewalld
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Next, we can install Docker and Docker Compose:

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io -y
sudo curl -L "https://github.com/docker/compose/releases/download/1.25.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

sudo systemctl enable docker
sudo systemctl start docker

Getting Harbor

You can deploy Harbor a number of different ways - today I'll be using Docker Compose to do so. Some may ask "Why not just deploy on your Kubernetes cluster?" - well, that's because your registry is best suited in a separate environment than a Kubernetes cluster that can be torn down and redeployed at a moment's notice.

Head on over to https://goharbor.io/ and grab the latest release.

wget https://github.com/goharbor/harbor/releases/download/v1.10.2/harbor-online-installer-v1.10.2.tgz
tar xvf harbor-online-installer-v1.10.2.tgz
sudo mv harbor /opt
cd /opt/harbor

## Create a directory for the container data and give it the right SELinux contexts
sudo mkdir /data
sudo chcon -Rt svirt_sandbox_file_t /data

SSL Certificates

Now, you could follow the Harbor docs and deploy your own self-signed certificates - I do in my lab. However, in this instance since we're working in the public cloud, we're going to use Let's Encrypt to generate SSL certificates for us.

sudo yum install certbot -y

certbot certonly --standalone --preferred-challenges http --non-interactive  --staple-ocsp --agree-tos -m you@example.com -d example.com

# Setup letsencrypt certificates renewing 
cat <<EOF > /etc/cron.d/letsencrypt
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
30 2 * * 1 root /usr/bin/certbot renew >> /var/log/letsencrypt-renew.log && cd /etc/letsencrypt/live/example.com && cp privkey.pem domain.key && cat cert.pem chain.pem > domain.crt && chmod 777 domain.*
EOF

# Rename SSL certificates
# https://community.letsencrypt.org/t/how-to-get-crt-and-key-files-from-i-just-have-pem-files/7348
cd /etc/letsencrypt/live/example.com && \
cp privkey.pem domain.key && \
cat cert.pem chain.pem > domain.crt

sudo mkdir -p /etc/docker/certs.d/example.com

sudo ln -s /etc/letsencrypt/live/example.com/chain.pem /etc/docker/certs.d/example.com/ca.crt

sudo ln -s /etc/letsencrypt/live/example.com/cert.pem /etc/docker/certs.d/example.com/example.com.cert

sudo ln -s /etc/letsencrypt/live/example.com/privkey.pem /etc/docker/certs.d/example.com/example.com.key

sudo systemctl restart docker

Deploy Harbor

Now that we have our SSL certificates in place, let's configure the harbor.yml file - ensure the following lines are changed:

hostname: EXAMPLE.com

https:
  port: 443
  # The path of cert and key files for nginx
  certificate: /etc/letsencrypt/live/example.com/fullchain.pem
  private_key: /etc/letsencrypt/live/example.com/privkey.pem

With that and whatever else you'd like changed configured, let's deploy the full Harbor suite:

sudo ./prepare
sudo ./install.sh --with-notary --with-clair --with-chartmuseum

With that, you should now be able to navigate to your hostname and access the Web UI - making sure to change the default admin password ASAP, of course.

Registry Replication

Now, while you already have a great working registry at hand, what if you're like me and you also need to replicate a private registry into a public one? Thankfully, replication is a first-class citizen in Harbor.

Harbor can replicate via a push or a pull action - in this case I'll be pushing from private into the public registry.

1. Public Harbor - Create a Project & Robot Account

First thing we have to do is navigate to our newly created Public Registry. We'll create a Project and in that project we'll create a Robot Account - this Robot Account will be the API key we need to interact with the registry.

Navigate to Projects
Click +New Project
Fill in the modal to your liking
Click the link of the new Project you created to enter it
Click the Robot Accounts tab
Click the + New Robot Account button
Fill in the modal as you'd like
Note the robot$YOUR_NAME - that's your Access ID, the long Token is your Access Secret. You'll need these two values in a moment to enable the Private registry to push into this Public registry.

2. Private Harbor - Create a Registry

Now switch over to the Private registry, we'll create the Registry entry and the Replication rule with that entry to push images into that Public registry.

Navigate to Administration > Registries
Click the + New Endpoint button
Fill in the modal with the details as you see needed for your Public registry - including the Access ID and Access Secret from the Robot Account in the Public Registry's Project you created earlier - whew.
Click Test Connection and OK if it pans out

3. Private Harbor - Create a Replication Rule

Navigate to Administration > Replications
Click the + New Replication Rule button
Fill out the modal as you'd like, referencing the newly created remote destination public registry we just defined. Note: I like to set the Trigger Mode to Event Based so the images will push automatically.
Click Save
Click the option circle to the left of your Replication rule you just created - click the Replicate button to run a manual replication

This can take some time depending on your network, how many repositories, how many tags, layers, and so on. A meager 22 artifacts at 1.7GB took about 16 minutes to push manually - the individual streams will be much faster though.

LDAP on GitLab with Red Hat Identity Management (FreeIPA)

Ken Moini — Tue, 28 Apr 2020 08:16:26 +0000

I have no idea why, but this weekend I redeployed everything I have in DigitalOcean - not permanently, just to try I guess...

Well, it did leave me with a lot of documentation at least! You'd be surprised how much stuff I just do and leave to the pages of history. This time was different, this time, I typed.

One of the things that has always gotten me is how to properly integrate LDAP into different things - every application has a different way of interacting and filtering the schema. GitLab is no exception, but today, I have finally mastered their thought of LDAP integration.

1. LDAP Setup

First off, I'll be using Red Hat Identity Management, or FreeIPA, as the LDAP server. Honestly it's LDAP with a bunch of other things but either way, there are a couple assumptions being made:

You have a gitlabusers Group in IDM/IPA
You have a gitlabadmins Group in IDM/IPA
You have a set of users assigned to those groups
You have a bind user dedicated to GitLab LDAP Binding

How do you make a dedicated Bind DN for your LDAP server? Make a file called gitlabbdn.update with the following contents:

dn: uid=gitlabbdn,cn=users,cn=accounts,dc=example,dc=com
add:objectclass:account
add:objectclass:simplesecurityobject
add:uid:gitlabbdn
add:userPassword:s3cr3tP455w0rdHERE
add:passwordExpirationTime:20380119031407Z
add:nsIdleTimeout:0

Then on your IPA server, as the admin user, run the following command:

ipa-ldap-updater gitlab-bind.update

Now you can use the Bind DN of uid=gitlabbdn,cn=users,cn=accounts,dc=example,dc=com and the s3cr3tP455w0rdHERE password you set to securely bind to the LDAP server.

2. GitLab Configuration

Next, we'll modify the LDAP section of the /etc/gitlab/gitlab.rb file to look something like this:

### LDAP Settings
###! Docs: https://docs.gitlab.com/omnibus/settings/ldap.html
###! **Be careful not to break the indentation in the ldap_servers block. It is
###!   in yaml format and the spaces must be retained. Using tabs will not work.**

gitlab_rails['ldap_enabled'] = true
gitlab_rails['prevent_ldap_sign_in'] = false

###! **remember to close this block with 'EOS' below**
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
  main:
    label: 'My LDAP'
    host: 'ipa.example.com'
    port: 389
    uid: 'uid'
    bind_dn: 'uid=gitlabbdn,cn=users,cn=accounts,dc=example,dc=com'
    password: 's3cr3tP455w0rdHERE'
    encryption: 'start_tls'
    verify_certificates: false
    smartcard_auth: false
    active_directory: false
    allow_username_or_email_login: false
    lowercase_usernames: false
    block_auto_created_users: false
    base: 'cn=accounts,dc=example,dc=com'
    user_filter: '(memberof=CN=gitlabusers,CN=groups,CN=accounts,DC=example,DC=com)'
    attributes:
      username: ['uid']
      email: ['mail']
      name: 'displayName'
      first_name: 'givenName'
      last_name: 'sn'
EOS

Outside of replacing the domain/credentials with yours, that should do it. Any LDAP user in the gitlabusers group will be able to access.

Run gitlab-ctl reconfigure and enjoy the new centralized authentication!

Lightweight DevOps Host with Red Hat Enterprise Linux

Ken Moini — Tue, 31 Mar 2020 03:27:02 +0000

I am exceptionally bad at this 100 Days of Code thing.

Not that I haven't been coding most of these days, but it just hasn't been fun, or sexy, or something to share. NO ONE WANTS TO READ ABOUT A DMARC MICROSERVICE, TRUST ME I'VE TRIED.

Anywho, here I am today in a new series that should hopefully be good for my own documentation as well. I'm in the process of deploying a new educational platform and it's all containerized, based on microservices deployed into a Kubernetes cluster, with sprinkles on top, and blah blah.

Thing is, I'm building this out in my home lab to avoid spending an extra hundred dollars in the cloud. So if you've got some hardware and time and want to do the same, I'll take you from Zero to Hero with this whole DevOps thing.

The Metal

So you can probably run this on anything really, but I'm going to start with a dedicated host and turn it into a container and virtualisation host. It's a simple Dell R620 with never enough Cores or RAM. The host sits on my internal network behind a pfSense router. It's got a static IP of 192.168.42.10 and we'll give it a hostname of thebus.kemo.labs.

Note that kemo.labs is not a valid TLD - that's ok. I have that domain resolved and routed via my pfSense router, you could do the same with DNSMASQ or BIND.

RHEL, RHEL, RHEL, we meet again

So this time I decided against going with Proxmox for the container/VM host to get closer to what my intended production environment will be like, which will be a Kubernetes/OpenShift cluster running CentOS/Red Hat Universal Basic Image (UBI) containers.

So some might want to just grab CentOS, which of course would work, but why not get your own free copy of Red Hat Enterprise Linux? How does one procure one of these free copies? Why, wouldn't you like to know...

Ok, so simply enough, one just needs to register with the Red Hat Developers site, and grab your very own yearly Red Hat Developer Suite subscription. It expires yearly, but it's still just a free renewal and gives you access to Red Hat Enterprise Linux, the Middleware offerings, and more.

So grab your subscription, download the ISO (I'll be using RHEL 7), and get it installed - so far that's the easy part.

Basic Provisioning

Now that we've got our Red Hat Enterprise Linux (RHEL) host set up, you're staring at the log in terminal, jumped into your privileged user, and are now at a shell. Let's bless this mess a scootch...

# Elevate to root
sudo -i

# In case you need to still register/attach a subscription
subscription-manager register
subscription-manager attach --auto-attach

# Enable repos
subscription-manager repos --enable=rhel-7-server-supplementary-rpms --enable=rhel-7-server-rpms --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-ansible-2.9-rpms --enable=rhel-7-server-extras-rpms

# Update system and reboot to pull in new kernels
yum update -y
systemctl reboot

# Install needed packages
yum install -y wget curl git nano ca-certificates cockpit cockpit-dashboard cockpit-docker cockpit-machines cockpit-packagekit cockpit-shell ansible docker docker-selinux libvirt-client

# Install Docker Compose
curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

# Set Firewall Options
firewall-cmd --add-service=ssh
firewall-cmd --add-service=cockpit

# Set services
systemctl enable cockpit && systemctl start cockpit
systemctl enable docker && systemctl start docker

So with alllll of that, we've done a few things:

Logged in, elevated to root
Used subscription-manage to register the system and attach your subscription.
Enabled the needed repositories, updated packages, and rebooted
Installed a few things:
- Base system packages such as curl, git, and nano
- Ansible, Docker, and the virsh (via libvirt-client)
- Cockpit and some modules for Cockpit - it's a Web UI that'll make your life sooooo much easier
Installed Docker Compose
Set firewall options for SSH access and Cockpit (exposed on port 9090)
Enabled and started Cockpit and Docker

And now with that, we have our own RHEL host that is ready to run a bunch of Containers, Virtual Machines, and Web UI with Cockpit to manage it all!

Join me in the next part of this series where I show how to install GitLab in a VM and Minio in a Docker container for local S3-compatible storage.

Access the OpenShift Internal Registry

Ken Moini — Thu, 13 Feb 2020 17:56:51 +0000

OpenShift has a really handy feature called Image Streams. It also has a built-in Docker V2 registry which is also very useful. However, the registry isn't really exposed outside the cluster. So how do you do rolling updates of images?

1. Ensure you have the needed RBAC policies

In order to push/pull from the internal OpenShift registry, your OpenShift user needs to have a few permissions. As either another cluster-admin user or via the Master OpenShift node in the system:admin user context, apply the following policies to your intented_user:

$ oc adm policy add-role-to-user system:registry <intended_user> -n <namespace/project>
$ oc adm policy add-role-to-user system:image-builder <intended_user> -n <namespace/project>
## or for cluster-wide access...
$ oc adm policy add-cluster-role-to-user system:registry <intended_user>
$ oc adm policy add-cluster-role-to-user system:image-builder <intended_user>

With those policies, you can access the internal registry with the intended_user and perform docker push/pull commands.

2. Log into your OpenShift application node

This is really the only trick to it - you don't want to do this on your Master nodes as that's in the system:admin context. You also can't do this on your local machine since it doesn't have access to the routed default.svc domain inside the cluster. So scoot on over to one of your infrastructure nodes and run the following:

$ oc login
$ sudo docker login -u openshift -p $(oc whoami -t) docker-registry.default.svc:5000

3. Pull n Push it Real Good

Now that we're logged into our internal OpenShift registry from an OpenShift application node, we can continue to push a new image to the Image Stream and Registry. When you have a Deployment Configuration that updates upon Image Change, this action will perform a Rolling Update as well.

$ sudo docker pull yourUser/someContainer:latest
$ sudo docker tag yourUser/someContainer:latest docker-registry.default.svc:5000/<namespace|project>/<image-stream-name>:<tag>
$ sudo docker push docker-registry.default.svc:5000/<namespace|project>/<image-stream-name>:<tag>

Now if your deployment is updated upon an Image Change you should see a rolling update happen very quicky - ultra-quick if you happen to be on the node that is running that pod since the image layers are already on the system from your docker pull command!

If only someone were able to make this into an Ansible playbook that could be fired off as part of a script locally or via Ansible Tower...

Automate GitLab with Ansible - kinda

Ken Moini — Mon, 10 Feb 2020 21:02:38 +0000

So there I was, automatin' and integratin' again. What can I say, it does get the best of me at times...

My task was to provide a way to automate the clean up of GitLab user namespaces, and import in a fresh repo from GitHub for NN student workshop users. Doesn't sound too hard eh?

So come to find out the GitLab module in Ansible is SUPER old and not really maintained. No problem, I'll just ping the GitHub API with Ansible.

Why do that instead of using something like Bash to perform cURL requests? Because the rest of the deployment already uses Ansible and it's much easier to just toss in a few extra tasks instead of bringing along an extra script and dropping into a different execution context.

1. Create the main Ansible Playbook

There's two Ansible files, and you'll see why shortly. First, let's create the configure_users.yaml file with the following content:

---
- name: Configure GitLab Workshop User Instances
  hosts: localhost
  gather_facts: false
  vars:
  - student_count: 50
  - github_personal_access_token: your_github_pat
  - github_repo: kenmoini/s2f-tasks-app
  - gitlab_endpoint: https://gitlab.example.com

  tasks:

  - name: Get GitHub Repo Import Repo ID
    uri:
      url: "https://api.github.com/repos/{{ github_repo }}"
      method: GET
      validate_certs: no
      status_code:
          - 200
          - 201
          - 409
    register: github_repo_info

  - name: Configure Per User GitLab Environment
    include_tasks: actual_user_config.yaml
    with_sequence: start=0 end="{{ student_count }}"

A few variables to keep an eye on, you need to pass in your GitHub Personal Access Token in order to import the repo, as well as the actual repo. We're also provisioning for 50 student user seats.
GitLab imports GitHub repos by their Repo ID, the numeric value, not the string. So our first task is to get that Repo ID
Next, we loop through an external tasks file - Blocks don't support loops.

2. Per-seat tasks

Let's create that external actual_user_config.yaml file:

---
- name: GitLab Post | Obtain Access Token
  uri:
    url: "{{ gitlab_endpoint }}/oauth/token"
    method: POST
    validate_certs: no
    status_code: 200
    body_format: json
    headers:
        Content-Type: application/json
    body: >
        {
        "grant_type": "password",
        "username": "student{{ item }}",
        "password": "user_password_here"
        }
  register: gitlab_access_token
  tags:
  - always

- name: GitLab Get | Get All User projects
  uri:
    url: "{{ gitlab_endpoint }}/api/v4/users/student{{ item }}/projects"
    method: GET
    validate_certs: no
    status_code:
        - 200
        - 201
        - 409
    headers:
        Content-Type: application/json
        Authorization: Bearer {{ gitlab_access_token.json.access_token }}
  register: user_projects
  tags:
  - destroy

- name: GitLab Post | Delete Projects via API
  uri:
    url: "{{ gitlab_endpoint }}/api/v4/projects/{{ project.id }}"
    method: DELETE
    validate_certs: no
    status_code:
        - 200
        - 201
        - 202
        - 409
    headers:
        Content-Type: application/json
        Authorization: Bearer {{ gitlab_access_token.json.access_token }}
  loop: "{{ user_projects.json }}"
  loop_control:
    loop_var: project
  tags:
  - destroy

- name: GitLab Post | Import Project from GitHub
  uri:
    url: "{{ gitlab_endpoint }}/api/v4/import/github"
    method: POST
    validate_certs: no
    status_code:
        - 200
        - 201
        - 409
        - 400
    body_format: json
    headers:
        Content-Type: application/json
        Authorization: Bearer {{ gitlab_access_token.json.access_token }}
    body: >
        {
        "personal_access_token": "{{ github_personal_access_token }}",
        "repo_id": "{{ github_repo_info.json.id }}",
        "target_namespace": "student{{ item }}"
        }
  tags:
  - import

This external set of tasks is executed top-to-bottom per-student.
First, we get an authentication token to use for the following tasks
Next, there are tagged tasks to get all the repos a user has access to
We can then destroy all the repos under the user
Once it's all cleaned up, we'll import a new repo fresh into each student namespace

3. Rolllll it

Now you can simply run one of the following commands:

ansible-playbook configure_users.yaml --tags destroy
# or
ansible-playbook configure_users.yaml --tags import
# OR OR
ansible-playbook configure_users.yaml --tags "destroy,import"

Hopefully, this helps in case you're looking to automate GitLab with Ansible. If there's not a particular Ansible module, or if it's not up to snuff then don't be afraid to use APIs if there's one available!

How to Diagnose OpenShift API Issues with More Verbose Logging

Ken Moini — Sun, 09 Feb 2020 04:34:40 +0000

So while I was integrating LDAP with OpenShift, I had a funny issue - it didn't work. The error message given at the Web UI was also useless, probably for the best, don't want to show how the sausage is made. On top of that, normal logging such as /var/log/{audit/*,messages,secure} showed nothing of these authentication failures.

Come to find out, the normal log-level of OpenShift is pretty low - understandably so, there are a metric TON of events going on in a larger Kubernetes system. So in order to find logging for LDAP and other authentication events, we have to bump up the log level a bit.

1. Download the master-loglevel.yml Ansible Playbook

Red Hat has a handy-dandy Ansible playbook that will modify and collect the log levels of your OpenShift masters!

$ wget https://access.redhat.com/sites/default/files/attachments/master-loglevel.yml

Moving on in the execution of this playbook, it's assumed that your host inventory is located at the default /etc/ansible/hosts location.

2. Increase the log

$ ansible-playbook master-loglevel.yml -t loglevel -e debug_loglevel=6

Now that we're logging at a loud enough volume, let's go and do things that break and don't work and stuff.

3. ?????

Really, at this point I'm not sure what you'd want to look for, but do the things that don't work. For me, I attempted logging into the LDAP users which generated an error which is now logged.

4. PROFIT!!!!1

Let's pull in those juicy logs:

$ ansible-playbook -i inventory master-loglevel.yml -t collect

Logs are saved under /tmp/master-logs/ of your ansible host, for each master .tar.gz.
It contains OpenShift Master API, Controllers and ETCD logs.

Odds are, you want to take a look at those OpenShift Master API logs.

5. Save dat cheddar (and log space)

Don't forget to turn off the logging to something more sane and less verbose...

$ ansible-playbook master-loglevel.yml -t loglevel -e debug_loglevel=2

Hopefully, this helps you track down some of those more system-level issues that you may face as an SRE!

Keystone, LDAP, and Multiple Identity Providers in OpenShift

Ken Moini — Sat, 08 Feb 2020 05:14:50 +0000

Today was momentous - I moved our OpenShift cluster and workloads off of AWS and onto the OROCK Cloud that runs Red Hat Cloud Suite top to bottom. Most of the workloads were easy to migrate and while I was shuffling things around I figured I'd deploy Red Hat Identity Management to the OROCK Cloud as well and get it integrated with our new OpenShift cluster.

A few notes for those who may not know much about the Red Hat ecosystem:

Red Hat Cloud Suite includes everything you need to run your own private cloud similar to AWS, Azure, GCP, etc
Red Hat OpenShift Container Platform is the enterprise Kubernetes platform that makes everyone's lives easier.
Red Hat Identity Management is the supported FreeIPA offering that wraps up a bunch of services such as DNS, LDAP, PKI, and more. I use it primarily for an LDAP store.
LDAP is similar to Active Directory in that it provides a hierarchal tree-based directory and authentication system.
Keystone is another enterprise authentication mechanism that is part of the OpenStack IaaS offering

Anywho, so since this OpenShift cluster is running in an OpenStack private cloud, it uses Keystone to authenticate my user. I use Red Hat IDM/LDAP for workshop user authentication because it's often easier to integrate into different solutions - and I don't want 100 student user accounts taking up space in Keystone next to my actual cluster-admin user.

So what I need are multiple authentication methods for OpenShift.

1. Pull in your IDM/LDAP CA

Odds are you're using a self-signed Certificate Authority certificate which means it's not in the normal keystores. We need a copy of that on our OpenShift Masters and we can pull it easily:

$ openssl s_client -connect idm.example.com -showcerts 2>/dev/null | sed --quiet '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | csplit --prefix=outfile - "/-----END CERTIFICATE-----/+1" "{*}" --elide-empty-files --quiet
$ sudo cp outfile01 /etc/ssl/certs/idm-ca.pem
$ sudo chmod 600 /etc/ssl/certs/idm-ca.pem

That will create two files, the last of which will likely be your CA certificate. Then the following commands copy it into place.

2. Modify the /etc/origin/master/master-config.yaml file

If you're rolling multi-masters, it's probably easier to modify the Ansible host file and modify the openshift_master_identity_providers variable and run the deployment playbooks, but for this example we'll modify the /etc/origin/master/master-config.yaml directly.

...
oauthConfig:
  identityProviders:
  - name: keystone
    challenge: true
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      domainName: exampleDoamin
      kind: KeystonePasswordIdentityProvider
      url: https://api.us-east-1.dacloud.com:13000/v3/
  - name: ldap
    challenge: true
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: LDAPPasswordIdentityProvider
      attributes:
        id:
        - dn
        email:
        - mail
        name:
        - cn
        preferredUsername:
        - uid
      bindDN: "uid=binddn,cn=accounts,dc=example,dc=com"
      bindPassword: "superSecretPass"
      ca: /etc/ssl/certs/idm-ca.pem
      insecure: false
      url: "ldaps://idm.example.com/cn=accounts,dc=example,dc=com?uid"
...

3. Restart the OpenShift Masters

That configuration manifest now allows the OpenShift cluster to authenticate with either the keystone method or the ldap method. Before it will take, all the masters need to be restarted, so run the following commands on your OpenShift Masters:

$ sudo /usr/local/bin/master-restart api && sudo /usr/local/bin/master-restart controllers

Once the masters restart, you should be able to log in to OpenShift with either identity providers now!

Cache Busting in Docker

Ken Moini — Tue, 28 Jan 2020 12:20:23 +0000

Just putting the finishing touches on a new Ansible Tower Workshop, testing an additional exercise, building my Docker container locally before I push the changes to the repo to build my production image in Docker Hub.

So as I'm sitting here in the Nashville Airport waiting for this container to build and a bit of time to pass for us to board, I figured it'd be a good time to do a bit of writing. Here's a trick I use that can save you hours in building containers...

The Premise

I start with a very basic Alpine image and add my needed packages - a few from the apk package manager, but then I compile nginx from source. This is because it's much easier to set proper operating conditions for the nginx server in a container. Then, after the nginx server is compiled and ready to roll, I build my application and dump it into /var/www/html

The Problem

Now, compiling nginx from source takes time. I have separate RUN stanzas in the Dockerfile to benefit from the image layering that container images provide, but Docker isn't always that great at detecting changes. If I make a few changes to the web application, Docker doesn't always copy over the new files.

To quickly fix this, one could prune the built images in the local cache - but this means starting from the top, which means recompiling nginx, which takes a good deal of time.

The Solution

How do we tell Docker explicitly where to start from? Just add an ENV between your static and dynamic actions, then change a small variable in between builds.

FROM node:8-alpine

# Install base packages
RUN apk -U add wget tar gzip git asciidoctor

# Install nginx
RUN \
  build_pkgs="build-base linux-headers openssl-dev pcre-dev wget zlib-dev" && \
  runtime_pkgs="ca-certificates openssl pcre zlib" && \
  apk --update add ${build_pkgs} ${runtime_pkgs} && \
  cd /tmp && \
  wget http://nginx.org/download/nginx-1.15.8.tar.gz && \
  tar xzf nginx-1.15.8.tar.gz && \
  cd /tmp/nginx-1.15.8 && \
  ./configure \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/tmp/run/nginx.pid \
    --lock-path=/tmp/run/nginx.lock \
    --http-client-body-temp-path=/tmp/cache/nginx/client_temp \
    --http-proxy-temp-path=/tmp/cache/nginx/proxy_temp \
    --http-fastcgi-temp-path=/tmp/cache/nginx/fastcgi_temp \
    --http-uwsgi-temp-path=/tmp/cache/nginx/uwsgi_temp \
    --http-scgi-temp-path=/tmp/cache/nginx/scgi_temp \
    --user=nginx \
    --group=nginx \
    --with-http_ssl_module \
    --with-http_realip_module \
    --with-http_addition_module \
    --with-http_sub_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_mp4_module \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_random_index_module \
    --with-http_secure_link_module \
    --with-http_stub_status_module \
    --with-http_auth_request_module \
    --with-mail \
    --with-mail_ssl_module \
    --with-file-aio \
    --with-ipv6 \
    --with-threads \
    --with-stream \
    --with-stream_ssl_module \
    --with-http_slice_module \
    --with-http_v2_module && \
  make && \
  make install && \
  ln -sf /dev/stdout /var/log/nginx/access.log && ln -sf /dev/stderr /var/log/nginx/error.log && \
  adduser -D nginx && \
  rm -rf /tmp/* && \
  apk del ${build_pkgs} && \
  rm -rf /var/cache/apk/* && \
  mkdir -p /tmp/cache/nginx/scgi_temp && \
  mkdir -p /tmp/cache/nginx/uwsgi_temp && \
  mkdir -p /tmp/cache/nginx/fastcgi_temp && \
  mkdir -p /tmp/cache/nginx/proxy_temp && \
  mkdir -p /tmp/cache/nginx/client_temp && \
  mkdir -p /tmp/run

ENV cacheBUSTER v9001

COPY conf/nginx.conf /etc/nginx/nginx.conf

COPY webapp/* /var/www/html/

Now, simply change that cacheBuster ENV definition to some other string and enjoy a cached image layer with nginx while being able to pull in fresh application data every time!

Automated DigitalOcean DNS ABCs: APIs, Bash, & cURL

Ken Moini — Wed, 15 Jan 2020 23:20:44 +0000

This script is an extraction of another larger single-node Kubernetes on DigitalOcean deployer that can be read here:

All-in-one Kubernetes Host on DigitalOcean

Ken Moini ・ Jan 13 '20

#kubernetes #devops #100daysofcode #beginners

So while doing the whole DevOps thing you usually have to build automation to deploy infrastructure and platforms. It's often not just Virtual Machines that you need to create and automate, but also DNS, networking, and more. When using Ansible and Terraform with the larger public cloud providers this is no problem, but sometimes the more niche providers such as DigitalOcean and Linode don't have mature modules and providers in those automation platforms.

Thankfully though, pretty much everyone has mature APIs available for use, including DigitalOcean and Linode! Yay, my friends are still cool! So in deploying a system to DigitalOcean with Ansible or Terraform we can create and use a Bash script wrapper that has some functions that call to the DigitalOcean API and process things for us.

So let's build out a Bash script that will:

Check to see if a DNS Zone exists for a supplied domain
Check to see if a specific record exists in that Zone
Create a record with a specific type and value
Add addition or delete and overwrite records when directed/forced

What-what-whaaaaat?

Beefy bits eh? I like providing the whole source upfront in case you wanna skip the small-talk, then stepping through the functionality to add detail. Let's review:

The commented set -x makes Bash print the invocation of each line as well, useful for debugging variables as they're expanded and passed around.
Set a few variables, the DO_PAT variable is imported when previously exported and set in the shell, otherwise is defined as blank. This is your DigitalOcean Personal Access Token, get one here.
Bash does not hoist functions like JavaScript does so functions needed earlier need to be defined before earlier. The print_help function is an example of that.
The if/while/switch-case defines and switches through arguments passed into the script, defines needed variables where supplied
Some arguments are required, so we check for $domain, $ip_addr, and $record_name
There are some programs that are required on the system this script is run, namely curl and jq - the Bash script has a checkForProgram function that will check for the existence of those programs
First up, the checkDomain function - this one is a pretty simple idea, before we do anything we want to make sure the domain in question has a Zone in DigitalOcean's DNS servers. We attempt to retrieve an existing domain from the DigitalOcean API and use jq to check to see if the returned JSON value is null or not
Next, we define the checkRecord function which takes in a supplied domain and record name+type. So say you wanted to check to see if the domain example.com has an A type record that is named k8s, or otherwise, that's to check if the A record for k8s.example.com exists or not.
Since you can store multiple records with the same name+type, we need to have a way to delete extra records that are stale. The deleteRecord function makes the API call to do exactly that with whatever DigitalOcean DNS Record ID it is passed.
Finally, the writeDNS function writes a DNS Record of the specified name, type, and value to a specific domain.

The rest of the script basically just goes through some logic to check if the domain exists, if so then check if the record exists - if not, create the specified record. There is also a forced override that will delete previous matched records and create the intended one.

Examples

Make sure to pass the executable bit to the script with chmod +x ./config_dns.sh

Before starting, take that DigitalOcean Personal Access Token and export it to a variable:

$ export DO_PAT="your-token-here"

1) Say we just used Ansible or Terraform to create a new Droplet. This new Droplet has an IP address of 12.34.56.78 and will be hosting our new blog at wombat-adventures.com and www.wombat-adventures.com. We can use this script like so:

$ ./config_dns.sh -d wombat-adventures.com \
                  -t "A" \
                  -r "@" \
                  -i "12.34.56.78"

$ ./config_dns.sh --domain wombat-adventures.com \
                  --type "A" \
                  --record "www" \
                  --ip "12.34.56.78"

2) You have a new Load Balancer with an IP Address of 98.76.54.43 and want to set a CNAME record at lb.internal.shootersbar.com, deleting any other records that match and ensuring only one exists with the new Load Balancer IP.

$ ./config_dns.sh -d shootersbar.com \
                  -t "CNAME" \
                  -r "lb.internal" \
                  -i "98.76.54.43" \
                  --force

3) You need to configure GMail/GSuite MX records for mail at hyperloop-sandwiches.net

$ ./config_dns.sh -d hyperloop-sandwiches.net \
                  -t "MX" \
                  -r "@" \
                  -i "ASPMX.L.GOOGLE.COM" \
                  -p "1" \
                  --force-add

$ ./config_dns.sh -d hyperloop-sandwiches.net \
                  -t "MX" \
                  -r "@" \
                  -i "ALT1.ASPMX.L.GOOGLE.COM" \
                  --priority "5" \
                  --force-add

$ ./config_dns.sh -d hyperloop-sandwiches.net \
                  -t "MX" \
                  -r "@" \
                  -i "ALT2.ASPMX.L.GOOGLE.COM" \
                  --priority "5" \
                  --force-add

$ ./config_dns.sh -d hyperloop-sandwiches.net \
                  -t "MX" \
                  -r "@" \
                  -i "ALT3.ASPMX.L.GOOGLE.COM" \
                  --priority "10" \
                  --force-add

$ ./config_dns.sh -d hyperloop-sandwiches.net \
                  -t "MX" \
                  -r "@" \
                  -i "ALT4.ASPMX.L.GOOGLE.COM" \
                  --priority "10" \
                  --force-add

That's that right there, pretty simple wrapper for the DigitalOcean API to set DNS records. It's fairly extensible, supports a wide range of records and use cases, and is pretty simple to plug into most pipelines and workflows so get to automatin'!