DEV Community: Baptiste Mille-Mathias

Openshift Z-patch upgrade in restricted environment without mirroring quay.io

Baptiste Mille-Mathias — Thu, 10 Sep 2020 14:27:29 +0000

Since version 4.2 Openshift features a disconnected installation method that permits to setup and upgrade a cluster that is not connected to internet.

This methods implies you prior copy the artifacts from the release you will install into your internal registry and to declare using an imagecontentsourcepolicy manifest the mirror so all url from quay.io will substitute on-the-fly by your internal.

On the day to day management this is not very convenient because it requires you to monitor and to copy when necessary, or to have a script regularly running and that will copy (even if you'll never use it) the latest release available.

However if you have a reverse-cache (or even a registry with reverse-cache like JFrog Artifactory setup for quay.io you can upgrade directly through it without having to mirror.

I assume you already setup for the remote-mirror so the following works

podman pull repository.mycorp.com:5000/openshift-release-dev/ocp-release@sha256:4841bd519be0d5c533d9b4da2996a6d0d340ae1eddb8a6ea213958c7460dcdb4

Create/Update an imagecontentsourcepolicy to point to your artifactory

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: artifactory-remote
spec:
  repositoryDigestMirrors:
  - mirrors:
    - repository.mycorp.com:5000/openshift-release-dev
    source: quay.io/openshift-release-dev

Now generate a digest of the images from a node that has access to internet.

#!/bin/bash
# TARGET_VERSION should be in format x.y.z, for example 4.3.28
export OCP_RELEASE_NUMBER=${TARGET_VERSION}
export DIGEST="$(oc adm release info quay.io/openshift-release-dev/ocp-release:${OCP_RELEASE_NUMBER}-x86_64 | sed -n 's/Pull From: .*@//p')"
export SIGNATURE_BASE64=$(curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/$(echo $DIGEST | cut -d: -f1)=$(echo $DIGEST | cut -d: -f2)/signature-1" | base64 -w0 && echo)
export DIGEST_ALGO=$(echo $DIGEST | cut -d: -f1)
export DIGEST_SIGNATURE=$(echo $DIGEST | cut -d: -f2)

cat <<EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
    name: release-image-${OCP_RELEASE_NUMBER}
    namespace: openshift-config-managed
    labels:
       release.openshift.io/verification-signatures: ""
binaryData:
    ${DIGEST_ALGO}-${DIGEST_SIGNATURE}: ${SIGNATURE_BASE64}
EOF

Now you are ready to use the upgrade procedure from redhat as if you had mirrored quay.io

Kubernetes Configuration Management with Ansible (Part 2)

Baptiste Mille-Mathias — Mon, 22 Jun 2020 19:03:34 +0000

Follow-up of this post. We explained the directory organization and the manifest files naming convention, now let's explore the ansible part.

Ansible was chosen because I knew it a bit since 2015 when I started automating some repetitive tasks to create a Kafka cluster prototype. Ansible is a good candidate because it can talk to HTTP API and it comes with a list of k8s related modules that make easy to query or manage Kubernetes objects (they were rewritten a couple of time before being now really good).

Overview of the Code

I don't plan to review all code, but have a look of key parts of the Ansible code.

The cluster accept a parameter which is the target cluster, then we load the common variables file then the cluster variable file from config directory

---
- name: "Deploy manifests on cluster {{ target_cluster }}"
  connection: local
  gather_facts: false
  vars:
    authorized_cluster: ["dev", "prod"]
  hosts: localhost

  pre_tasks:
    - name: Check if the target_cluster variable was properly set
      fail:
        msg: "You need to specify the variable target_cluster with one value ({{ authorized_cluster }})"
      when: target_cluster not in authorized_cluster

    - name: Load common variables
      include_vars:
        file: configs/common.yml

    - name: "Load cluster-scoped variables for {{ target_cluster }}"
      include_vars:
        file: "config/{{ target_cluster }}.yml"

for the given cluster, we create a list of all yaml files from the common and $cluster directories. We will also define some variables than will be used later

---
- name: "Retrieve manifests files from common and {{ cluster }} directories"
  find:
    paths: ["{{ yaml_dir }}/common", "{{ yaml_dir }}/{{ cluster }}"]
    patterns: ['*.yaml', '*.yml']
    file_type: file
    recurse: true
  register: manifests_list

- name: "Define variable from manifests filename"
  vars:
    filename: "{{ item.path | basename }}"
    kind: "{{ item.path | basename | regex_replace('.*_(.*)_.*_.*_.*\\.ya?ml', '\\1') }}"
    name: "{{ item.path | basename | regex_replace('.*_.*_(.*)_.*_.*\\.ya?ml', '\\1') }}"
    namespace: "{{ item.path | basename | regex_replace('.*_.*_.*_(.*)_.*\\.ya?ml', '\\1') }}"
    state: "{{ item.path | basename | regex_replace('.*_.*_.*_.*_(.*)\\.ya?ml', '\\1') }}"

  set_fact:
    manifests_information: "{{ manifests_information + [{ 'filepath': item.path, 'filename': filename, 'kind': kind, 'name': name, 'namespace': namespace, 'state': state }] }}"
  loop: "{{ manifests_list.files | sort(attribute='path') }}"
  loop_control:
    label: "{{ filename }}"

Now we check each file honors the filename convention else we throw an error.

- name: "Check files match the naming convention"
  fail:
    msg: >
      The file {{ item.filepath }} does not observe the manifest naming convention.
  when: item.filename == item.kind or
    item.filename == item.name or
    item.filename == item.namespace or
    item.state not in [ "present", "absent" ]
  with_items:
    - "{{ manifests_information }}"
  loop_control:
    label: "{{ item.filepath }}"

for each file, call the k8s module, the path of of the file is passed to argument using the lookup plugin template, so the manifest can have logic and variables inside.

---
- name: "Deploy {{ item.kind }} {{ item.name }} on namespace {{ item.namespace }} to {{ item.state }}"
  k8s:
    host: "https://xxxxxx:6443"
    api_key: "{{ token }}"
    state: "{{ item.state }}"
    definition: "{{ lookup ('template', item.filepath) }}"

For the code we're done, for a stripped down version it does not require much than that to work.

Coding your Manifests

One of the feature this small playbook permits is to have simple manifest yaml file like this

---
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  name: "{{ cluster-admin-ns }}"
...

but you can also using feature like Ansible variables and Jinja templating together.
For instance, I had to put a config file into a secret, which require to have the content base64 encoded. It's convenient when you can do that automatically.

In this example, the content for the key fluent.conf come from a file fluent.conf.j2 loaded by the template lookup (so the variable are replaced) then passed to the filter b64encode.

---
apiVersion: v1
data:
  fluent.conf: "{{ lookup('template', 'fluent.conf.j2') | b64encode }}"
kind: secret
metadata:
  name: fluent-forwarder-secret
  namespace: logging
...

with the configuration file treated as a template

# fluent.conf.j2
# As this file is interpreted as a ansible template, you can add
# jinja code inside to put some logic.

<source>
  @type  forward
  port  24224
  tag logs.openshift
</source>

<match **>
  # Send all types to Splunk
  @type splunk_hec
  hec_host {{ splunk.host }}
  hec_port {{ splunk.port }}
  hec_token {{ splunk.token }}
  index {{ splunk.index }}
</match>

so it gives once loaded something like

apiVersion: v1
data:
  fluent.conf: IyBmbHVlbnQuY29uZi5qMgojIEFzIHRoaXMgZmlsZSBpcyBpbnRlcnByZXRlZCBhcyBhIGFuc2libGUgdGVtcGxhdGUsIHlvdSBjYW4gYWRkCiMgamluamEgY29kZSBpbnNpZGUgdG8gcHV0IHNvbWUgbG9naWMuCgo8c291cmNlPgogIEB0eXBlICBmb3J3YXJkCiAgcG9ydCAgMjQyMjQKICB0YWcgbG9ncy5vcGVuc2hpZnQKPC9zb3VyY2U+Cgo8bWF0Y2ggKio+CiAgIyBTZW5kIGFsbCB0eXBlcyB0byBTcGx1bmsKICBAdHlwZSBzcGx1bmtfaGVjCiAgaGVjX2hvc3Qgc3BsdW5rLmRvbS50bGQKICBoZWNfcG9ydCA4MDg5CiAgaGVjX3Rva2VuIDM1NDM0My0zNDUzMzMtMTUyMzMKICBpbmRleCBmYWtlLWluZGV4CjwvbWF0Y2g+
kind: Secret

... Just what expect the secret API.
So I think we're good for now, I just wanted to give you an idea of what it is possible to do. This is a skeleton that can be enhanced with more check and feature.

Perhaps I'll come back with new idea.

Kubernetes Configuration Management with Ansible (Part 1)

Baptiste Mille-Mathias — Mon, 22 Jun 2020 16:52:04 +0000

Since day 0 I started to administrate Openshift last year, one of the core thing I wanted we tackle with the team was how to manage all customization we would have to do. I'm used to automation software, from bare shell scripts (yes this is automation :) to things like Puppet, CFengine or Ansible and I can't imagine now managing an application, a fleet of nodes, a cluster without being able to automate the deployment of the configuration.

So I share here - not in details - what we did in our team. I'll split the explanation in a couple of posts. This is quite simple but effective, we started recently to run regularly this job from AWX in check mode to observe configuration drift and may run it in run mode sooner or later.

Set a plan

We started to make some status & requirements with the colleagues (this is ALWAYS important to do so)

We will administrate several clusters.
A big part of the clusters configuration will be the same for all clusters.
We must be able to deploy for each cluster some specific manifests
The same manifests will deployed on several cluster but may differ slightly (quota values for instance), and we don't want to store all these manifests just for the sake of different values inside. So we must be able have placeholder in the manifests which would be filled with each cluster values.
We must be able to add manifests but also to remove manifests from the cluster.

Directory structure

Here is an example of the structure we choose

project-dir
├── config
│   ├── common.yml
│   ├── dev.yml
│   └── prod.yml
└── manifests
    ├── common
    │   ├── 05_authentication
    │   ├── 10_feature_foo
    │   └── 30_feature_baz
    ├── dev
    │   ├── 40_feature_dev_buz
    │   └── 60_feature_dev_buu
    └── prod
        ├── 50_feature_prod_bar
        └── 70_feature_prod_boo

Directory config contains a yaml file for each cluster, with the connection details and all specific variables, the file common.yml contains common and default variables.

# prod config file
connection:
  - url: https://prd.prj.domain.tld
    token: !vault....

ldap_connection:
  - bind_user: cn=ldap-prd-user,...
    bind_password: !vault...

authorized_groups:
  -  group-dev-1
  -  group-dev-2
  -  group-dev-3

# other prod specific variables

File naming Convention

We think after about a naming convention for the manifest files, not that it would enforce the content in the file (exception made to the status field), but to permit to easily spot a manifest file on the disk. We came up with convention XX_kind_name_namespace_status.yml where

XX is a number to which help applying manifest in a specific order.
kind is the kind of the manifest (deployment, configMap, ...).
name is the value of the file metadata.name of the object to be modified.
namespace is the namespace the object belongs to. If the object is not namespaced, like namespace or clusterrole, we set the value to global.
status has either value present to ensure it exists or absent to ensure it does not exist. This value in the name of the file will be to the module k8s and will enforce the state of the manifest.

few examples

To better understand how it works, let have a look

a manifest file to create a namespace foobar would be named 10_namespace_foobar_global_present.yml
a deployment named fluentd in namespace cluster-logging would be named 30_deployment_fluentd_cluster-logging_present.yml
a manifest to delete a configMap named my-config in namespace dev-hideout would be named XX_configmap_my-config_dev-hideouts_absent.yml

Now we had an overview of the structure of the project, the next post will explain the ansible playbook.