DEV Community: Mikhail Khomenko

InterSystems Kubernetes Operator Deep Dive: Part 2

Mikhail Khomenko — Sat, 13 Mar 2021 11:00:35 +0000

In the previous article, we looked at one way to create a custom operator that manages the IRIS instance state. This time, we’re going to take a look at a ready-to-go operator, InterSystems Kubernetes Operator (IKO). Official documentation will help us navigate the deployment steps.

Prerequisites

To deploy IRIS, we need a Kubernetes cluster. In this example, we’ll use Google Kubernetes Engine (GKE), so we’ll need to use a Google account, set up a Google Cloud project, and install gcloud and kubectl command line utilities.

You’ll also need to install the Helm3 utility:

$ helm version
version.BuildInfo{Version:"v3.3.4"...}

Note: Be aware that on Google free tier, not all resources are free.

It doesn’t matter in our case which type of GKE we use – zonal, regional, or private. After we create one, let’s connect to the cluster. We’ve created a cluster called "iko" in a project called "iko-project". Use your own project name in place of "iko-project" in the later text.

This command adds this cluster to our local clusters configuration:

$ gcloud container clusters get-credentials iko --zone europe-west2-b --project iko-project

Install IKO

Let’s deploy IKO into our newly-created cluster. The recommended way to install packages to Kubernetes is using Helm. IKO is not an exception and can be installed as a Helm chart. Choose Helm version 3 as it's more secure.

Download IKO from the WRC page InterSystems Components, creating a free developer account if you do not already have one. At the moment of writing, the latest version is 2.0.223.0.

Download the archive and unpack it. We will refer to the unpacked directory as the current directory.

The chart is in the chart/iris-operator directory. If you just deploy this chart, you will receive an error when describing deployed pods:

Failed to pull image "intersystems/iris-operator:2.0.0.223.0": rpc error:
code = Unknown desc = Error response from daemon:
pull access denied for intersystems/iris-operator,
repository does not exist or may require 'docker login'.

So, you need to make an IKO image available from the Kubernetes cluster. Let’s push this image into Google Container Registry first:

$ docker load -i image/iris_operator-2.0.0.223.0-docker.tgz
$ docker tag intersystems/iris-operator:2.0.0.223.0 eu.gcr.io/iko-project/iris-operator:2.0.0.223.0
$ docker push eu.gcr.io/iko-project/iris-operator:2.0.0.223.0

After that, we need to direct IKO to use this new image. You should do this by editing the Helm values file:

$ vi chart/iris-operator/values.yaml
...
operator:
  registry: eu.gcr.io/iko-project
...

Now, we’re ready to deploy IKO into GKE:

$ helm upgrade iko chart/iris-operator --install --namespace iko --create-namespace

$ helm ls --all-namespaces --output json | jq '.[].status'
"deployed"

$ kubectl -n iko get pods # Should be Running with Readiness 1/1

Let’s look at the IKO logs:

$ kubectl -n iko logs -f --tail 100 -l app=iris-operator
…
I1212 17:10:38.119363 1 secure_serving.go:116] Serving securely on [::]:8443
I1212 17:10:38.122306 1 operator.go:77] Starting Iris operator

Custom Resource Definition irisclusters.intersystems.com was created during IKO deployment.
You can look at the API schema it supports, although it is quite long:

$ kubectl get crd irisclusters.intersystems.com -oyaml | less

One way to look at all available parameters is to use the "explain" command:

$ kubectl explain irisclusters.intersystems.com

Another way is using jq. For instance, viewing all top-level configuration settings:

$ kubectl get crd irisclusters.intersystems.com -ojson | jq '.spec.versions[].schema.openAPIV3Schema.properties.spec.properties | to_entries[] | .key'
"configSource"
"licenseKeySecret"
"passwordHash"
"serviceTemplate"
"topology"

Using jq in this way (viewing the configuration fields and their properties), we can find out the following configuration structure:

configSource
  name
licenseKeySecret
  name
passwordHash
serviceTemplate
  metadata
    annotations
  spec
    clusterIP
    externalIPs
    externalTrafficPolicy
    healthCheckNodePort
    loadBalancerIP
    loadBalancerSourceRanges
    ports
    type
topology
  arbiter
    image
    podTemplate
      controller
        annotations
      metadata
        annotations
      spec
        affinity
          nodeAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
          podAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
          podAntiAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
        args
        env
        imagePullSecrets
        initContainers
        lifecycle
        livenessProbe
        nodeSelector
        priority
        priorityClassName
        readinessProbe
        resources
        schedulerName
        securityContext
        serviceAccountName
        tolerations
    preferredZones
    updateStrategy
      rollingUpdate
      type    
  compute
    image
    podTemplate
      controller
        annotations
      metadata
        annotations
      spec
        affinity
          nodeAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
          podAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
          podAntiAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
        args
        env
        imagePullSecrets
        initContainers
        lifecycle
        livenessProbe
        nodeSelector
        priority
        priorityClassName
        readinessProbe
        resources
          limits
          requests
        schedulerName
        securityContext
        serviceAccountName
        tolerations
    preferredZones
    replicas
    storage
      accessModes
      dataSource
        apiGroup
        kind
        name
      resources
        limits
        requests
      selector
      storageClassName
      volumeMode
      volumeName
    updateStrategy
      rollingUpdate
      type
  data
    image
    mirrored
    podTemplate
      controller
        annotations
      metadata
        annotations
      spec
        affinity
          nodeAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
          podAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
          podAntiAffinity
            preferredDuringSchedulingIgnoredDuringExecution
            requiredDuringSchedulingIgnoredDuringExecution
        args
        env
        imagePullSecrets
        initContainers
        lifecycle
        livenessProbe
        nodeSelector
        priority
        priorityClassName
        readinessProbe
        resources
          limits
          requests
        schedulerName
        securityContext
        serviceAccountName
        tolerations
    preferredZones
    shards
    storage
      accessModes
      dataSource
        apiGroup
        kind
        name
      resources
        limits
        requests
      selector
      storageClassName
      volumeMode
      volumeName
    updateStrategy
      rollingUpdate
      type

There are so many settings, but, you don’t need to set them all. The defaults are suitable. You can see examples of configuration in the file iris_operator-2.0.0.223.0/samples.

To run a minimal viable IRIS, we need to specify only a few settings, like IRIS (or IRIS-based application) version, storage size, and license key.

Note about license key: we’ll use a community IRIS, so we don’t need a key. We cannot just omit this setting, but can create a secret containing a pseudo-license. License secret generation is simple:


 $ touch iris.key # remember that a real license file is used in the most cases
 $ kubectl create secret generic iris-license --from-file=iris.key

An IRIS description understandable by IKO is:

$ cat iko.yaml
apiVersion: intersystems.com/v1alpha1
kind: IrisCluster
metadata:
  name: iko-test
spec:
  passwordHash: '' # use a default password SYS
  licenseKeySecret:
    name: iris-license # use a Secret name bolded above
  topology:
    data:
      image: intersystemsdc/iris-community:2020.4.0.524.0-zpm # Take a community IRIS
      storage:
        resources:
          requests:
            storage: 10Gi

Send this manifest into the cluster:

$ kubectl apply -f iko.yaml

$ kubectl get iriscluster
NAME       DATA COMPUTE MIRRORED STATUS   AGE
iko-test 1                       Creating 76s

$ kubectl -n iko logs -f --tail 100 -l app=iris-operator
db.Spec.Topology.Data.Shards = 0
I1219 15:55:57.989032 1 iriscluster.go:39] Sync/Add/Update for IrisCluster default/iko-test
I1219 15:55:58.016618 1 service.go:19] Creating Service default/iris-svc.
I1219 15:55:58.051228 1 service.go:19] Creating Service default/iko-test.
I1219 15:55:58.216363 1 statefulset.go:22] Creating StatefulSet default/iko-test-data.

We see that some resources (Service, StatefulSet) are going to be created in a cluster in the "default" namespace.

In a few seconds, you should see an IRIS pod in the "default" namespace:

$ kubectl get po -w
NAME            READY STATUS             RESTARTS AGE
iko-test-data-0 0/1   ContainerCreating  0        2m10s

Wait a little until the IRIS image is pulled, that is, until Status becomes Ready and Ready becomes 1/1. You can check what type of disk was created:

$ kubectl get pv
NAME                                     CAPACITY ACCESS MODES  RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-b356a943-219e-4685-9140-d911dea4c106 10Gi     RWO    Delete Bound   default/iris-data-iko-test-data-0 standard 5m

Reclaim policy "Delete" means that when you remove Persistent Volume, GCE persistent disk will be also removed. There is another policy, "Retain", that allows you to save Google persistent disks to survive Kubernetes Persistent Volumes deletion. You can define a custom StorageClass to use this policy and other non-default settings. An example is present in IKO’s documentation: Create a storage class for persistent storage.

Now, let’s check our newly created IRIS. In general, traffic to pods goes through Services or Ingresses. By default, IKO creates a service of ClusterIP type with a name from the iko.yaml metadata.name field:

$ kubectl get svc iko-test
NAME     TYPE      CLUSTER-IP EXTERNAL-IP PORT(S)             AGE
iko-test ClusterIP 10.40.6.33 <none>      1972/TCP,52773/TCP  14m

We can call this service using port-forward:

$ kubectl port-forward svc/iko-test 52773

Navigate a browser to http://localhost:52773/csp/sys/UtilHome.csp and type _system/SYS.

You should see a familiar IRIS user interface (UI).

Custom Application

Let’s replace a pure IRIS with an IRIS-based application. First, download the COVID-19 application. We won’t consider a complete, continuous deployment here, just minimal steps:

$ git clone https://github.com/intersystems-community/covid-19.git
$ cd covid-19
$ docker build --no-cache -t covid-19:v1 .

As our Kubernetes is running in a Google cloud, let’s use Google Docker Container Registry as an image storage. We assume here that you have an account in Google Cloud allowing you to push images. Use your own project name in the below-mentioned commands:

$ docker tag covid-19:v1 eu.gcr.io/iko-project/covid-19:v1
$ docker push eu.gcr.io/iko-project/covid-19:v1

Let’s go to the directory with iko.yaml, change the image there, and redeploy it. You should consider removing the previous example first:

$ cat iko.yaml
...  
  data:
    image: eu.gcr.io/iko-project/covid-19:v1
...
$ kubectl delete -f iko.yaml
$ kubectl -n iko delete deploy -l app=iris-operator
$ kubectl delete pvc iris-data-iko-test-data-0
$ kubectl apply -f iko.yaml

You should recreate the IRIS pod with this new image.

This time, let’s provide external access via Ingress Resource. To make it work, we should deploy an Ingress Controller (choose nginx for its flexibility). To provide a traffic encryption (TLS), we will also add yet another component – cert-manager.

To install both these components, we use a Helm tool, version 3.

$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

$ helm upgrade nginx-ingress  \
  --namespace nginx-ingress   \
  ingress-nginx/ingress-nginx \
  --install                   \
  --atomic                    \
  --version 3.7.0             \
  --create-namespace

Look at an nginx service IP (it’s dynamic, but you can make it static):


 $ kubectl -n nginx-ingress get svc
 NAME                                   TYPE         CLUSTER-IP   EXTERNAL-IP PORT(S) AGE
 nginx-ingress-ingress-nginx-controller LoadBalancer 10.40.0.103  xx.xx.xx.xx 80:32032/TCP,443:32374/TCP 88s

Note: your IP will differ.

Go to your domain registrar and create a domain name for this IP. For instance, create an A-record:

covid19.myardyas.club = xx.xx.xx.xx

Some time will pass until this new record propagates across DNS servers. The end result should be similar to:

$ dig +short covid19.myardyas.club
xx.xx.xx.xx

Having deployed Ingress Controller, we now need to create an Ingress resource itself (use your own domain name):

$ cat ingress.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: iko-test
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    certmanager.k8s.io/cluster-issuer: lets-encrypt-production # Cert manager will be deployed below
spec:
  rules:
  - host: covid19.myardyas.club
    http:
      paths:
      - backend:
          serviceName: iko-test
          servicePort: 52773
        path: /
  tls:
  - hosts:
    - covid19.myardyas.club
    secretName: covid19.myardyas.club

$ kubectl apply -f ingress.yaml

After a minute or so, IRIS should be available at http://covid19.myardyas.club/csp/sys/UtilHome.csp (remember to use your domain name) and the COVID-19 application at http://covid19.myardyas.club/dsw/index.html (choose namespace IRISAPP).

Note: Above, we’ve exposed the HTTP IRIS port. If you need to expose via nginx TCP super-server port (1972 or 51773), you can read instructions at Exposing TCP and UDP services.

Add Traffic Encryption

The last step is to add traffic encryption. Let’s deploy cert-manager for that:

$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.10.0/deploy/manifests/00-crds.yaml

$ helm upgrade cert-manager \
  --namespace cert-manager  \
  jetstack/cert-manager     \
  --install                 \
  --atomic                  \
  --version v0.10.0         \
  --create-namespace

$ cat lets-encrypt-production.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: lets-encrypt-production
spec:
  acme:
    # Set your email. Let’s Encrypt will send notifications about certificates expiration
    email: mvhoma@gmail.com 
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: lets-encrypt-production
    solvers:
    - http01:
        ingress:
          class: nginx

$ kubectl apply -f lets-encrypt-production.yaml

Wait a few minutes until cert-manager notices IRIS-application ingress and goes to Let’s Encrypt for a certificate. You can observe Order and Certificate resources in progress:

$ kubectl get order
NAME                             STATE AGE
covid19.myardyas.club-3970469834 valid 52s

$ kubectl get certificate
NAME                  READY SECRET                AGE
covid19.myardyas.club True  covid19.myardyas.club 73s

This time, you can visit a more secured site version - https://covid19.myardyas.club/dsw/index.html:

About Native Google Ingress Controller and Managed Certificates

Google supports its own ingress controller, GCE, which you can use in place of an nginx controller. However, it has some drawbacks, for instance, lack of rewrite rules support, at least at the moment of writing.

Also, you can use Google managed certificates in place of cert-manager. It’s handy, but initial retrieval of certificate and any updates of Ingress resources (like new path) causes a tangible downtime. Also, Google managed certificates work only with GCE, not with nginx, as noted in Managed Certificates.

Next Steps

We’ve deployed an IRIS-based application into the GKE cluster. To expose it to the Internet, we’ve added Ingress Controller and a certification manager. We’ve tried the IrisCluster configuration to highlight that setting up IKO is simple. You can read about more settings in Using the InterSystems Kubernetes Operator documentation.

A single data server is good, but the real fun begins when we add ECP, mirroring, and monitoring, which are also available with IKO. Stay tuned and read the upcoming article in our Kubernetes operator series to take a closer look at mirroring.

InterSystems Kubernetes Operator Deep Dive: Introduction to Kubernetes Operators

Mikhail Khomenko — Thu, 21 Jan 2021 09:05:52 +0000

Introduction

Several resources tell us how to run IRIS in a Kubernetes cluster, such as Deploying an InterSystems IRIS Solution on EKS using GitHub Actions and Deploying InterSystems IRIS solution on GKE Using GitHub Actions. These methods work but they require that you create Kubernetes manifests and Helm charts, which might be rather time-consuming.
To simplify IRIS deployment, InterSystems developed an amazing tool called InterSystems Kubernetes Operator (IKO). A number of official resources explain IKO usage in details, such as New Video: Intersystems IRIS Kubernetes Operator and InterSystems Kubernetes Operator.

Kubernetes documentation says that operators replace a human operator who knows how to deal with complex systems in Kubernetes. They provide system settings in the form of custom resources. An operator includes a custom controller that reads these settings and performs steps the settings define to correctly set up and maintain your application. The custom controller is a simple pod deployed in Kubernetes. So, generally speaking, all you need to do to make an operator work is deploy a controller pod and define its settings in custom resources.
You can find high-level explanation of operators in How to explain Kubernetes Operators in plain English. Also, a free O’Reilly ebook is available for download.
In this article, we’ll have a closer look at what operators are and what makes them tick. We’ll also write our own operator.

Prerequisites and Setup

To follow along, you’ll need to install the following tools:
kind

$ kind --version
kind version 0.9.0

golang

$ go version
go version go1.13.3 linux/amd64

kubebuilder

$ kubebuilder version
Version: version.Version{KubeBuilderVersion:"2.3.1"…

kubectl

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11"...

operator-sdk

$ operator-sdk version
operator-sdk version: "v1.2.0"…

Custom Resources

API resources is an important concept in Kubernetes. These resources enable you to interact with Kubernetes via HTTP endpoints that can be grouped and versioned. The standard API can be extended with custom resources, which require that you provide a Custom Resource Definition (CRD). Have a look at the Extend the Kubernetes API with CustomResourceDefinitions page for detailed info.
Here is an example of a CRD:

$ cat crd.yaml 
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: irises.example.com
spec:
  group: example.com
  version: v1alpha1
  scope: Namespaced
  names:
    plural: irises
    singular: iris
    kind: Iris
    shortNames:
    - ir
  validation:
    openAPIV3Schema:
      required: ["spec"]
      properties:
        spec:
          required: ["replicas"]
          properties:
            replicas:
              type: "integer"
              minimum: 0

In the above example, we define the API GVK (Group/Version/Kind) resource as example.com/v1alpha1/Iris, with replicas as the only required field.
Now let’s define a custom resource based on our CRD:

$ cat crd-object.yaml 
apiVersion: example.com/v1alpha1
kind: Iris
metadata:
  name: iris
spec:
  test: 42
  replicas: 1

In our custom resource, we can define any fields in addition to replicas, which is required by the CRD.
After we deploy the above two files, our custom resource should become visible to standard kubectl.
Let’s launch Kubernetes locally using kind, and then run the following kubectl commands:

$ kind create cluster
$ kubectl apply -f crd.yaml
$ kubectl get crd irises.example.com
NAME                 CREATED AT
irises.example.com   2020-11-14T11:48:56Z

$ kubectl apply -f crd-object.yaml
$ kubectl get iris
NAME   AGE
iris   84s

Although we’ve set a replica amount for our IRIS, nothing actually happens at the moment. It’s expected. We need to deploy a controller - the entity that can read our custom resource and perform some settings-based actions.
For now, let’s clean up what we’ve created:

$ kubectl delete -f crd-object.yaml
$ kubectl delete -f crd.yaml

Controller

A controller can be written in any language. We’ll use Golang as Kubernetes’ "native" language. We could write a controller’s logic from scratch but the good folks from Google and RedHat gave us a leg up. They have created two projects that can generate the operator code that will only require minimum changes – kubebuilder and operator-sdk. These two are compared at the kubebuilder vs operator-sdk page, as well as here: What is the difference between kubebuilder and operator-sdk #1758.

Kubebuilder

It is convenient to start our acquaintance with Kubebuilder at the Kubebuilder book page. The Tutorial: Zero to Operator in 90 minutes video from the Kubebuilder maintainer might help as well.

Sample implementations of the Kubebuilder project can be found in the sample-controller-kubebuilder and in kubebuilder-sample-controller repositories.

Let’s scaffold a new operator project:

$ mkdir iris
$ cd iris
$ go mod init iris # Creates a new module, name it iris
$ kubebuilder init --domain myardyas.club # An arbitrary domain, used below as a suffix in the API group

Scaffolding includes many files and manifests. The main.go file, for instance, is the entrypoint of code. It imports the controller-runtime library, instantiates and runs a special manager that keeps track of the controller run. Nothing to change in any of these files.

Let’s create the CRD:

$ kubebuilder create api --group test --version v1alpha1 --kind Iris
Create Resource [y/n]
y
Create Controller [y/n]
y
…

Again, a lot of files are generated. These are described in detail at the Adding a new API page. For example, you can see that a file for kind Iris is added in api/v1alpha1/iris_types.go. In our first sample CRD, we defined the required replicas field. Let’s create an identical field here, this time in the IrisSpec structure. We’ll also add the DeploymentName field. The replicas’ count should be also visible in the Status section, so we need to make the following changes:

$ vim api/v1alpha1/iris_types.go
…
type IrisSpec struct {
        // +kubebuilder:validation:MaxLength=64
        DeploymentName string `json:"deploymentName"`
        // +kubebuilder:validation:Minimum=0
        Replicas *int32 `json:"replicas"`
}
…
type IrisStatus struct {
        ReadyReplicas int32 `json:"readyReplicas"`
}
…

After editing the API, we’ll move to editing the controller boilerplate. All the logic should be defined in the Reconcile method (this example is mostly taken from mykind_controller.go). We also add a couple of auxiliary methods and rewrite the SetupWithManager method.


$ vim controllers/iris_controller.go
…
import (
...
// Leave the existing imports and add these packages
        apps "k8s.io/api/apps/v1"
        core "k8s.io/api/core/v1"
        apierrors "k8s.io/apimachinery/pkg/api/errors"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/client-go/tools/record"
)
// Add the Recorder field to enable Kubernetes events
type IrisReconciler struct {
        client.Client
        Log    logr.Logger
        Scheme *runtime.Scheme
        Recorder record.EventRecorder
}
…
// +kubebuilder:rbac:groups=test.myardyas.club,resources=iris,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=test.myardyas.club,resources=iris/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch

func (r *IrisReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("iris", req.NamespacedName)
    // Fetch Iris objects by name
    log.Info("fetching Iris resource")
    iris := testv1alpha1.Iris{}
    if err := r.Get(ctx, req.NamespacedName, &iris); err != nil {
        log.Error(err, "unable to fetch Iris resource")
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    if err := r.cleanupOwnedResources(ctx, log, &iris); err != nil {
        log.Error(err, "failed to clean up old Deployment resources for Iris")
        return ctrl.Result{}, err
    }

    log = log.WithValues("deployment_name", iris.Spec.DeploymentName)
    log.Info("checking if an existing Deployment exists for this resource")
    deployment := apps.Deployment{}
    err := r.Get(ctx, client.ObjectKey{Namespace: iris.Namespace, Name: iris.Spec.DeploymentName}, &deployment)
    if apierrors.IsNotFound(err) {
        log.Info("could not find existing Deployment for Iris, creating one...")

        deployment = *buildDeployment(iris)
        if err := r.Client.Create(ctx, &deployment); err != nil {
            log.Error(err, "failed to create Deployment resource")
            return ctrl.Result{}, err
        }

        r.Recorder.Eventf(&iris, core.EventTypeNormal, "Created", "Created deployment %q", deployment.Name)
        log.Info("created Deployment resource for Iris")
        return ctrl.Result{}, nil
    }
    if err != nil {
        log.Error(err, "failed to get Deployment for Iris resource")
        return ctrl.Result{}, err
    }

    log.Info("existing Deployment resource already exists for Iris, checking replica count")

    expectedReplicas := int32(1)
    if iris.Spec.Replicas != nil {
        expectedReplicas = *iris.Spec.Replicas
    }
    
    if *deployment.Spec.Replicas != expectedReplicas {
        log.Info("updating replica count", "old_count", *deployment.Spec.Replicas, "new_count", expectedReplicas)            
        deployment.Spec.Replicas = &expectedReplicas
        if err := r.Client.Update(ctx, &deployment); err != nil {
            log.Error(err, "failed to Deployment update replica count")
            return ctrl.Result{}, err
        }

        r.Recorder.Eventf(&iris, core.EventTypeNormal, "Scaled", "Scaled deployment %q to %d replicas", deployment.Name, expectedReplicas)

        return ctrl.Result{}, nil
    }

    log.Info("replica count up to date", "replica_count", *deployment.Spec.Replicas)
    log.Info("updating Iris resource status")
    
    iris.Status.ReadyReplicas = deployment.Status.ReadyReplicas
    if r.Client.Status().Update(ctx, &iris); err != nil {
        log.Error(err, "failed to update Iris status")
        return ctrl.Result{}, err
    }

    log.Info("resource status synced")
    return ctrl.Result{}, nil
}

// Delete the deployment resources that no longer match the iris.spec.deploymentName field
func (r *IrisReconciler) cleanupOwnedResources(ctx context.Context, log logr.Logger, iris *testv1alpha1.Iris) error {
    log.Info("looking for existing Deployments for Iris resource")

    var deployments apps.DeploymentList
    if err := r.List(ctx, &deployments, client.InNamespace(iris.Namespace), client.MatchingField(deploymentOwnerKey, iris.Name)); err != nil {
        return err
    }
    
    deleted := 0
    for _, depl := range deployments.Items {
        if depl.Name == iris.Spec.DeploymentName {
            // Leave Deployment if its name matches the one in the Iris resource
            continue
        }

        if err := r.Client.Delete(ctx, &depl); err != nil {
            log.Error(err, "failed to delete Deployment resource")
            return err
        }

        r.Recorder.Eventf(iris, core.EventTypeNormal, "Deleted", "Deleted deployment %q", depl.Name)
        deleted++
    }

    log.Info("finished cleaning up old Deployment resources", "number_deleted", deleted)
    return nil
}

func buildDeployment(iris testv1alpha1.Iris) *apps.Deployment {
    deployment := apps.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:            iris.Spec.DeploymentName,
            Namespace:       iris.Namespace,
            OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(&iris, testv1alpha1.GroupVersion.WithKind("Iris"))},
        },
        Spec: apps.DeploymentSpec{
            Replicas: iris.Spec.Replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{
                    "iris/deployment-name": iris.Spec.DeploymentName,
                },
            },
            Template: core.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{
                        "iris/deployment-name": iris.Spec.DeploymentName,
                    },
                },
                Spec: core.PodSpec{
                    Containers: []core.Container{
                        {
                            Name:  "iris",
                            Image: "store/intersystems/iris-community:2020.4.0.524.0",
                        },
                    },
                },
            },
        },
    }
    return &deployment
}

var (
    deploymentOwnerKey = ".metadata.controller"
)

// Specifies how the controller is built to watch a CR and other resources 
// that are owned and managed by that controller
func (r *IrisReconciler) SetupWithManager(mgr ctrl.Manager) error {
    if err := mgr.GetFieldIndexer().IndexField(&apps.Deployment{}, deploymentOwnerKey, func(rawObj runtime.Object) []string {
        // grab the Deployment object, extract the owner...
        depl := rawObj.(*apps.Deployment)
        owner := metav1.GetControllerOf(depl)
        if owner == nil {
            return nil
        }
        // ...make sure it's an Iris...
        if owner.APIVersion != testv1alpha1.GroupVersion.String() || owner.Kind != "Iris" {
            return nil
        }

        // ...and if so, return it
        return []string{owner.Name}
    }); err != nil {
        return err
    }

    return ctrl.NewControllerManagedBy(mgr).
        For(&testv1alpha1.Iris{}).
        Owns(&apps.Deployment{}).
        Complete(r)
}

To make the events logging work, we need to add yet another line to the main.go file:


if err = (&controllers.IrisReconciler{
                Client: mgr.GetClient(),
                Log:    ctrl.Log.WithName("controllers").WithName("Iris"),
                Scheme: mgr.GetScheme(),
                Recorder: mgr.GetEventRecorderFor("iris-controller"),
        }).SetupWithManager(mgr); err != nil {

Now everything is ready to set up an operator.
Let’s install the CRD first using the Makefile target install:

$ cat Makefile
…
# Install CRDs into a cluster
install: manifests
        kustomize build config/crd | kubectl apply -f -
...
$ make install

You can have a look at the resulting CRD YAML file in the config/crd/bases/ directory.
Now check CRD existence in the cluster:

$ kubectl get crd
NAME                      CREATED AT
iris.test.myardyas.club   2020-11-17T11:02:02Z

Let’s run our controller in another terminal, locally (not in Kubernetes) – just to see if it actually works:

$ make run
...
2020-11-17T13:02:35.649+0200 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2020-11-17T13:02:35.650+0200 INFO setup starting manager
2020-11-17T13:02:35.651+0200 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-11-17T13:02:35.752+0200 INFO controller-runtime.controller Starting EventSource {"controller": "iris", "source": "kind source: /, Kind="}
2020-11-17T13:02:35.852+0200 INFO controller-runtime.controller Starting EventSource {"controller": "iris", "source": "kind source: /, Kind="}
2020-11-17T13:02:35.853+0200 INFO controller-runtime.controller Starting Controller {"controller": "iris"}
2020-11-17T13:02:35.853+0200 INFO controller-runtime.controller Starting workers {"controller": "iris", "worker count": 1}
...

Now that we have the CRD and the controller installed, all we need to do is create an instance of our custom resource. A template can be found in the config/samples/example.com_v1alpha1_iris.yaml file. In this file, we need to make changes similar to those in the crd-object.yaml:

$ cat config/samples/test_v1alpha1_iris.yaml
apiVersion: test.myardyas.club/v1alpha1
kind: Iris
metadata:
  name: iris
spec:
  deploymentName: iris
  replicas: 1

$ kubectl apply -f config/samples/test_v1alpha1_iris.yaml

After a brief delay caused by the need to pull an IRIS image, you should see the running IRIS pod:

$ kubectl get deploy
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
iris   1/1     1            1           119s

$ kubectl get pod
NAME                   READY   STATUS    RESTARTS   AGE
iris-6b78cbb67-vk2gq   1/1     Running   0          2m42s

$ kubectl logs -f -l iris/deployment-name=iris

You can open the IRIS portal using the kubectl port-forward command:

$ kubectl port-forward deploy/iris 52773

Go to http://localhost:52773/csp/sys/UtilHome.csp in your browser.
What if we change the replicas’ count in CRD? Let’s make and apply this change:

$ vi config/samples/test_v1alpha1_iris.yaml
…
  replicas: 2
$ kubectl apply -f config/samples/test_v1alpha1_iris.yaml

You should now see another Iris pod appear.

$ kubectl get events
...
54s         Normal   Scaled                    iris/iris                   Scaled deployment "iris" to 2 replicas
54s         Normal   ScalingReplicaSet         deployment/iris             Scaled up replica set iris-6b78cbb67 to 2

Log messages in the terminal where the controller in running report successful reconciliation:

2020-11-17T13:09:04.102+0200 INFO controllers.Iris replica count up to date {"iris": "default/iris", "deployment_name": "iris", "replica_count": 2}
2020-11-17T13:09:04.102+0200 INFO controllers.Iris updating Iris resource status {"iris": "default/iris", "deployment_name": "iris"}
2020-11-17T13:09:04.104+0200 INFO controllers.Iris resource status synced {"iris": "default/iris", "deployment_name": "iris"}
2020-11-17T13:09:04.104+0200 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "iris", "request": "default/iris"}

Okay, our controllers seem to be working. Now we’re ready to deploy that controller inside Kubernetes as a pod. For that, we need to create the controller docker container and push it to the registry. This can be any registry that works with Kubernetes – DockerHub, ECR, GCR, and so on.
We’ll use the local (kind) Kubernetes, so let’s deploy the controller to the local registry using the kind-with-registry.sh script available from the Local Registry page. We can simply remove the current cluster and recreate it:

$ kind delete cluster
$ ./kind_with_registry.sh
$ make install
$ docker build . -t localhost:5000/iris-operator:v0.1 # Dockerfile is autogenerated by kubebuilder
$ docker push localhost:5000/iris-operator:v0.1
$ make deploy IMG=localhost:5000/iris-operator:v0.1

The controller will be deployed into the IRIS-system namespace. Alternatively, you can scan all pods to find a namespace like kubectl get pod -A):

$ kubectl -n iris-system get po
NAME                                      READY   STATUS    RESTARTS   AGE
iris-controller-manager-bf9fd5855-kbklt   2/2     Running   0          54s

Let’s check the logs:

$ kubectl -n iris-system logs -f -l control-plane=controller-manager -c manager

You can experiment with changing replicas’ count in the CRD and observe how these changes are reflected in the IRIS instances count.

Operator-SDK

Another handy tool to generate the operator code is Operator SDK. To get the initial idea of this tool, have a look at this tutorial. You should install operator-sdk first.
For our simple use case, the process will look similar to the one we’ve worked on with kubebuilder (you can delete/create the kind cluster with the Docker registry before continuing). Run in another directory:

$ mkdir iris
$ cd iris
$ go mod init iris
$ operator-sdk init --domain=myardyas.club
$ operator-sdk create api --group=test --version=v1alpha1 --kind=Iris
# Answer two ‘yes’

Now change the IrisSpec and IrisStatus structures in the same file – api/v1alpha1/iris_types.go.
We’ll use the same iris_controller.go file as we did in kubebuilder. Don’t forget to add the Recorder field in the main.go file.
Because kubebuilder and operator-sdk use different versions of the Golang packages, you should add a context in the SetupWithManager function in controllers/iris_controller.go:

ctx := context.Background()
if err := mgr.GetFieldIndexer().IndexField(ctx, &apps.Deployment{}, deploymentOwnerKey, func(rawObj runtime.Object) []string {

Then, install the CRD and the operator (make sure that the kind cluster is running):

$ make install
$ docker build . -t localhost:5000/iris-operator:v0.2
$ docker push localhost:5000/iris-operator:v0.2
$ make deploy IMG=localhost:5000/iris-operator:v0.2

You should now see the CRD, operator pod, and IRIS pod(s) similar to the ones we’ve seen when we worked with kubebuilder.

Conclusion

Although a controller includes a lot of code, you’ve seen that changing the IRIS replicas is just a matter of changing a line in a custom resource. All the complexity is hidden in the controller implementation. We’ve looked at how a simple operator can be created using handy scaffolding tools.
Our operator cared only about IRIS replicas. Now imagine that we actually need to have the IRIS data persisted on disk – this would require StatefulSet and Persistent Volumes. Also, we would need a Service and, perhaps, Ingress for external access. We should be able to set the IRIS version and system password, Mirroring and/or ECP, and so on. You can imagine the amount of work InterSystems had to do to simplify IRIS deployment by hiding all the IRIS-specific logic inside operator code.
In the next article, we’re going to look at IRIS Operator (IKO) in more detail and investigate its possibilities in more complex scenarios.

Creating custom SNMP OIDs in InterSystems IRIS

Mikhail Khomenko — Fri, 16 Oct 2020 12:18:27 +0000

This post is dedicated to the task of monitoring aa IRIS instance using SNMP. Some users of IRIS are probably doing it already in some way or another. Monitoring via SNMP has been supported by the standard IRIS package for a long time now, but not all the necessary parameters are available "out of the box". For example, it would be nice to monitor the number of CSP sessions, get detailed information about the use of the license, particular KPI’s of the system being used and such. After reading this article, you will know how to add your parameters to IRIS monitoring using SNMP.

What we already have

IRIS can be monitored using SNMP. A full list of what’s supported can be found in the files of the <Install_dir>/SNMP/. You should find a file called ISC-IRIS.mib there. In particular, we’d like to know what information we can get about licenses and sessions. The table contains corresponding OID’s provided that the hierarchy starts from the root for InterSystems— 1.3.6.1.4.1.16563

OID	Name	Description	Data type
.4.1.1.1.10	irisSysLicenseUsed	The current number of licenses used on this IRIS instance	INTEGER
.4.1.1.1.11	irisSysLicenseHigh	The high-water mark for licenses used on this IRIS instance	INTEGER
.4.2.15	irisLicenseExceed	A request for a license has exceeded the licenses available or allowed	Trap message
.4.1.1.1.6	irisSysCurUser	Current number of users on this IRIS instance	INTEGER

The package lacks many important parameters, such as, for instance, the number of CSP sessions, license information and, of course, does not have application-specific KPI’s.

Here is an example of what we’d like to know:

The number of CSP users
Limitations of our license in terms of the user count
License expiry date

Let’s also add a few parameters for performance analysis. The parameters themselves are in the package, but we want to know the increment per minute, for example:

The increase of the number of "global" references per minute
The number of executed command per minute
The number of routine calls per minute

How to add "your" parameters

You can rely on the Monitoring InterSystems IRIS Using SNMP document.

The IRIS version of our test instance (IRIS) is 2020.3.0.200.0com. The operating system is Ubuntu 18.04.4 LTS (docker image intersystemsdc/iris-community:2020.3.0.200.0-zpm is used, root user is enabled).

Here is our agenda:

Create a class for collecting metrics.
Register and activate a new class in IRIS using ^%SYSMONMGR.
Create a user MIB using MonitorTools.SNMP class methods. We’ll use 99990 as a temporary PEN (Private Enterprise Number), but will need to register with IANA afterwards. This procedure is free, takes a week or two and requires some email exchange along the lines of "what do you need your own PEN for?".
Start a monitoring service with a connected IRIS subagent.
Use snmpwalk to make sure we have access to all our newly-created OID’s.
Add our OID’s to to a third-party monitoring system. Let’s use Zabbix, for example. Zabbix documentation is available here. Let’s make sure that monitoring is up and running.
Add the start of the system monitor in our TEST namespace to the system startup list.

Let’s now follow the agenda, point by point.

1. Create a class for collecting metrics

The metrics collection class extends %Monitor.Adaptor. In the Terminal we switch to the %SYS namespace and export the hidden Monitor.Sample class:

%SYS>do $system.OBJ.Export("Monitor.Sample.cls","/tmp/Monitor_Sample.xml")
Exporting to XML started on 10/16/2020 09:33:55
Exporting class: Monitor.Sample
Export finished successfully.

Let’s assume that the TEST namespace is our working area. Let’s switch to it and import the Monitor.Sample class here:

TEST>do $system.OBJ.Load("/tmp/Monitor_Sample.xml", "ck")

Now, we create a class that describes the implementation of a monitoring mechanism for the 6 metrics described in the "What we already have" section.

Class monitoring.snmp.Metrics Extends %Monitor.Adaptor
{
/// Give the application a name. This allows you to group different
/// classes together under the same application level in the SNMP MIB.
/// The default is the same as the Package name.
Parameter APPLICATION = "Monitoring";
/// CSP sessions count
Property Sessions As %Monitor.Integer;
/// License user limit
Property KeyLicenseUnits As %Monitor.Integer;
/// License key expiration date
Property KeyExpirationDate As %Monitor.String;
/// Global references speed
Property GloRefSpeed As %Monitor.Integer;
/// Number of commands executed
Property ExecutedSpeed As %Monitor.Integer;
/// Number of routine loads/save
Property RoutineLoadSpeed As %Monitor.Integer;
/// The method is REQUIRED. It is where the Application Monitor
/// calls to collect data samples, which then get picked up by the
/// ^SNMP server process when requested.
Method GetSample() As %Status
{
      set ..Sessions = ..getSessions()
      set ..KeyLicenseUnits = ..getKeyLicenseUnits()
      set ..KeyExpirationDate = ..getKeyExpirationDate()

      set perfList = ..getPerformance()
      set ..GloRefSpeed = $listget(perfList,1)
      set ..ExecutedSpeed = $listget(perfList,2)
      set ..RoutineLoadSpeed = $listget(perfList,3)

      quit $$$OK
}
/// Get CSP sessions count
Method getSessions() As %Integer
{
     // This method will only work if we don't use WebAddon:
    // quit $system.License.CSPUsers() 
    //
    // This will work even if  we use WebAddon:
    set csp = ""
    try {
        set cn = $NAMESPACE
        znspace "%SYS"
        set db = ##class(SYS.Stats.Dashboard).Sample()
        set csp = db.CSPSessions
        znspace cn
    } catch e {
        set csp = "0"
    }
    quit csp
}
/// Get license user's power
Method getKeyLicenseUnits() As %Integer
{
    quit $system.License.KeyLicenseUnits()
}
/// Get license expiration date in human-readable format
Method getKeyExpirationDate() As %String
{
    quit $zdate($system.License.KeyExpirationDate(),3)
}
/// Get performance metrics (gloref, rourines etc.)
Method getPerformance(param As %String) As %Integer
{
    set cn = $NAMESPACE
    znspace "%SYS"
    set m = ##class(SYS.Monitor.SystemSensors).%New()
    do m.GetSensors()
    znspace cn
    quit $listbuild(m.SensorReading("GlobalRefsPerMin"), 
                    m.SensorReading("RoutineCommandsPerMin"), 
                    m.SensorReading("RoutineLoadsPerMin"))
}
}

Make sure that the GetSample() method really fetches the necessary data for us:

TEST>set metrics = ##class(monitoring.snmp.Metrics).%New()

TEST>write metrics.GetSample()
1
TEST>zwrite metrics 
metrics=3@monitoring.snmp.Metrics  ; <OREF>
+----------------- general information ---------------
|      oref value: 3
|      class name: monitoring.snmp.Metrics
| reference count: 2
+----------------- attribute values ------------------
|      ExecutedSpeed = 2653596
|        GloRefSpeed = 35863
|  KeyExpirationDate = "2021-10-30"
|    KeyLicenseUnits = 5
|   RoutineLoadSpeed = 38
|           Sessions = 5
+-----------------------------------------------------

2. Let’s register and activate the new class in IRIS using ^%SYSMONMGR

Open the terminal and switch to the TEST namespace:

# iris session iris -U TEST
TEST>do ^%SYSMONMGR
1. Select item 5, Manage Application Monitor.

2. Select item 2, Manage Monitor Classes.

3. Select item 3, Register Monitor System Classes.
Exporting to XML started on 10/16/2020 10:05:39
Exporting class: Monitor.Sample
Export finished successfully.

Load started on 10/16/2020 10:05:39
Loading file /usr/irissys/mgr/Temp/VonCEUzQ8gWgfQ.stream as xml
Imported class: Monitor.Sample, using worker jobs
Compiling class Monitor.Sample
Compiling table Monitor.Sample
Compiling routine Monitor.Sample.1
Load finished successfully.

4. Select item 1, Activate/Deactivate Monitor Class
Class??
Num MetricsClassName Activated
1) %Monitor.System.AuditCount N
…
15) monitoring.snmp.Metrics N
Class? 15 monitoring.snmp.Metrics
Activate class? Yes => Yes

5. Select item 7, Exit

6. Select item 6, Exit

7. Select item 1, Start/Stop System Monitor

8. Select item 2, Stop System Monitor
Stopping System Monitor… System Monitor not running!

9. Select item 1, Start System Monitor
Starting System Monitor… System Monitor started

10. Select item 3, Exit

11. Select item 4, View System Monitor State
Component                     State
System Monitor                     OK
%SYS.Monitor.AppMonSensor          OK

12. Select item 7, Exit

3. Create a user MIB

A user MIB is created with the help of MonitorTools.SNMP class methods. For this example, let’s use a fake PEN (Private Enterprise Number), 99990, but the PEN will have to be registered with IANA afterwards. You can view registered numbers here. For example, InterSystems’ PEN is 16563.

16563
InterSystems
Robert Davis
rdavis&intersystems.com

We will use the MonitorTools.SNMP class and its CreateMIB() method to create a MIB file. This method takes 10 arguments:

Argument name and type	Description	Value
AppName As %String	application name	Value of the APPLICATION parameter of the metrics.snmp.Metrics class
Namespace As %String	our namespace	TEST
EntID As %Integer	company PEN	99990 (fiction)
AppID As %Integer	application OID inside the company	42
Company As %String	company name (capital letters)	fiction
Prefix As %String	prefix of all SNMP objects we create	fiction
CompanyShort As %String	short company prefix (capital letters)	fict
MIBname As %String	name of the MIB file	ISC-TEST
Contact As %String	contact information (address, in particular)	Let’s leave the default value: Earth, Russia, Somewhere in the forests, Subject: ISC-TEST.mib
List As %Boolean	equivalent to verbose. Show task progress for the MIB file	1

And here comes the creation of the MIB file:

%SYS>do ##class(MonitorTools.SNMP).CreateMIB("Monitoring","TEST",99990,42,"fiction","fict","fiction","ISC-TEST",,1)
Create SNMP structure for Application - Monitoring
     Group - Metrics
          ExecutedSpeed = Integer
          GloRefSpeed = Integer
          KeyExpirationDate = String
          KeyLicenseUnits = Integer
          RoutineLoadSpeed = Integer
          Sessions = Integer

Create MIB file for Monitoring
     Generate table Metrics
          Add object ExecutedSpeed, Type = Integer
          Add object GloRefSpeed, Type = Integer
          Add object KeyExpirationDate, Type = String
          Add object KeyLicenseUnits, Type = Integer
          Add object RoutineLoadSpeed, Type = Integer
          Add object Sessions, Type = Integer
MIB done.

There is new MIB ISC-TEST.mib in the <Install_dir>/mgr/TEST folder now.

4. Start the monitoring service with the connected IRIS subagent

Let’s open the System Administration -> Security -> Services -> %Service_Monitor (click) -> Service Enabled (check)

We also specify that we want to start the SNMP subagent when IRIS is started:

In Linux, we use the net-snmp package for SNMP monitoring. So we install it, configure it to be used with subagents and specify port 705 as the default one for the master agent to talk with subagents.
A small article about the snmpd.conf configuration file that complements the manual can be found on cyberciti. Here is your final set of settings:

# apt-get update
# apt-get install -y snmpd snmp
# grep '^[^#]' /etc/snmp/snmpd.conf
master agentx
agentXSocket TCP:localhost:705
com2sec local localhost public
group MyRWGroup v1 local
group MyRWGroup v2c local
group MyRWGroup usm local
view all included .1 80
view system included .iso.org.dod
access MyROGroup "" any noauth exact all none none
access MyRWGroup "" any noauth exact all all none
syslocation server (edit /etc/snmp/snmpd.conf)
syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
dontLogTCPWrappersConnects yes

Let’s restart the snmpd and snmptrapd daemons in Linux. After that, we start the SNMP service to activate the SNMP IRIS subagent:

%SYS>do start^SNMP

%SYS>; Check SNMP subagent status

%SYS>zwrite ^SYS("MONITOR")
^SYS("MONITOR","SNMP")="RUN"
^SYS("MONITOR","SNMP","NAMESPACE")="%SYS"
^SYS("MONITOR","SNMP","PID")=2035
^SYS("MONITOR","SNMP","PORT")=705
^SYS("MONITOR","SNMP","STARTUP")="SNMP agent started on port 705, timeout=20, winflag=0, Debug=0"
^SYS("MONITOR","SNMP","STATE")="Terminated - 10/16/2020 10:40:31.7147AM"
^SYS("MONITOR","SNMP","WINSTART")=0

5. Check that only our own, newly-created user OID’s are available

This can be done using snmpwalk — we’ll display the OID showing the number of CSP sessions:

# snmpwalk -On -v 2c -c public localhost 1.3.6.1.4.1.99990
.1.3.6.1.4.1.99990.42.1.1.1.1.4.73.82.73.83 = INTEGER: 1928761
.1.3.6.1.4.1.99990.42.1.1.1.2.4.73.82.73.83 = INTEGER: 226351
.1.3.6.1.4.1.99990.42.1.1.1.3.4.73.82.73.83 = STRING: "2021-10-30"
.1.3.6.1.4.1.99990.42.1.1.1.4.4.73.82.73.83 = INTEGER: 5
.1.3.6.1.4.1.99990.42.1.1.1.5.4.73.82.73.83 = INTEGER: 306
.1.3.6.1.4.1.99990.42.1.1.1.6.4.73.82.73.83 = INTEGER: 2

# If you get such result
# .1.3.6.1.4.1.99990 = No Such Object available on this agent at this OID
# try to restart SNMP subagent in IRIS in this way:
# do stop^SNMP
# do start^SNMP

The ISC-TEST.mib file contains the sequence of our OID’s:

FictMetricsR ::=
       SEQUENCE {
        fictExecutedSpeed   Integer32,
        fictGloRefSpeed   Integer32,
        fictKeyExpirationDate   DisplayString,
        fictKeyLicenseUnits   Integer32,
        fictRoutineLoadSpeed   Integer32,
        fictSessions   Integer32
       }

Accordingly, the number of sessions, for example, is the last OID 1.3.6.1.4.1.99990.42.1.1.1.6. You can compare it with the number of sessions shown on the SMP dashboard:

6. Let’s add our OID’s to an external monitoring system

Let’s use Zabbix. Zabbix documentation can be found here. A detailed Linux installation and configuration guide for Zabbix is available here. Zabbix was selected as a system that not only allows you to draw charts, but also monitor Plain Text (in our case, license expiry date and license units). After adding our 6 metrics to our local host items (type: SNMPv2 agent) and creating 4 graphs and 2 PlainText parameters (as screen elements), we should see the following picture:

Above is the information about license expiry and the number of available license slots. Graphs speak for themselves.

7. Let’s add the launch of the system monitor to the startup list of our TEST namespace

There is a pretty good document about user routines executed when IRIS starts and stops. They are called %ZSTART and %ZSTOP, accordingly.

What we are interested in is that the system monitor (^%SYSMONMGR) starts in the TEST namespace during the system start. By default, this monitor only starts on the %SYS namespace. Therefore, we will only look at the ^%ZSTART program. The source is in %ZSTART.mac (create and save it to the %SYS namespace).

ROUTINE %ZSTART

%ZSTART; User startup routine.
SYSTEM;
    ; IRIS starting
    do $zu(9,"","Starting System Monitor in TEST namespace by ^%ZSTART...Begin")
    znspace "TEST"
    set sc = ##class(%SYS.Monitor).Start()
    do $system.OBJ.DisplayError(sc)
    if (sc = 1) {
    do $zutil(9,"","Starting System Monitor in TEST namespace by ^%ZSTART...OK")
    } else {
    do $zutil(9,"","Starting System Monitor in TEST namespace by ^%ZSTART...ERROR")
    }
    ; Starting SNMP
    znspace "%SYS"
    do start^SNMP
    quit
LOGIN;
    ; a user logs into IRIS (user account or telnet)
    quit
JOB;
    ; JOB'd process begins
    quit
CALLIN;
    ; a process enters via CALLIN interface
    quit

Another way to do the same is using ^%SYSMONMGR:

%SYS>do ^%SYSMONMGR

1. Select item 3, Configure System Monitor Classes.

2. Select item 2, Configure Startup Namespaces.

3. Select item 2, Add Namespace.
Namespace? TEST

4. Select item 1, List Start Namespaces.
Option? 1
     TEST

5. Select item 4, Exit.

6. Select item 3, Exit.

7. Select item 8, Exit.

Let’s now restart IRIS (if possible) to make sure that SNMP stats continue to be collected after a restart.

This is it. Perhaps, some will question my choice of monitored parameters or code, but the task was to show the mere possibility of implementing such monitoring in principle. You can add extra parameters or refactor your code later.
Also, it's worth to make IRIS settings persistent in Docker using Installer, but it's not in a scope of this article.

Adding TLS and DNS to IRIS-based Services Deployed on Google Kubernetes Engine

Mikhail Khomenko — Fri, 02 Oct 2020 06:27:12 +0000

This article is a continuation of Deploying InterSystems IRIS solution on GKE Using GitHub Actions, in which, with the help of GitHub Actions pipeline, our zpm-registry was deployed in a Google Kubernetes cluster created by Terraform. In order not to repeat, we’ll take as a starting point that:

You’ve forked the repository Github Actions + GKE + zpm example and allow Actions in your fork. Its root directory will be referenced as <root_repo_dir> throughout the article.
You’ve replaced the placeholders in the Terraform file.
You’ve created all the secrets (in the GitHub Actions Secrets page) mentioned in the only table in Deploying InterSystems IRIS solution on GKE Using GitHub Actions.
All code samples will be stored in the GitHub-GKE-TLS repository to simplify copying and pasting.

Assuming all the above, let’s continue.

Getting Started

Last time we made our connection to zpm-registry like this:

curl -XGET -u _system:SYS 104.199.6.32:52773/registry/packages/-/all

The absence of a protocol before the IP address means that we’re using HTTP and, therefore, traffic is unencrypted and Infamous Eve can catch our password.

To protect against Eve eavesdropping, we need to encrypt the traffic, which means using HTTPS. Can we do this?

104.199.6.32:52773/registry/packages → https://104.199.6.32:52773/registry/packages

Generally speaking, yes. You can obtain the certificate for an IP address. But this solution has drawbacks. Read Using an IP Address in an SSL Certificate and SSL for IP Address for more details. In short, you need a static IP address and there are no free certificate providers with that ability. The benefit is also questionable.

As for free providers in general, fortunately, they do exist, as you’ll find in 9 Best Free SSL Certificate Sources. One of the leaders is Let’s Encrypt, but it issues certificates only for domain names, although there are plans to add IP Addresses in Certificates.

So, to encrypt the traffic, you first need to get a domain name. If you already have a domain name, you can skip the next section.

Getting a Domain Name

Buy a domain name from a domain registrar. There are quite a few. Prices depend on the domain name and level of service. For development purposes, you can take, for example, a domain with the suffix .dev, which will probably be inexpensive.

We won’t get into the domain registration process here, there’s nothing complicated about it. Note that from here on we’ll assume the registered domain name is example.com. Substitute your own domain.

At the end of the domain registration process, you’ll have a domain name and domain zone. They are not the same thing. See the note Definition - Domains vs. Zones and this small video, DNS Zones, to understand the difference.

Briefly, if you imagine the domain as an organization, the zone can be thought of as a department in this organization. Our domain is example.com and we’ll call the zone the same: example.com. (In a small organization you can have just one department.) Each DNS zone should have special servers that know about the IP addresses and domain names inside its zone. These links are called resource records (RR). They can be of different types (see DNS Record types). The most widely used is the A-type.

The special servers, called name servers, are provided by the domain registrar. For example, my registrar gave me the following two name servers:

ns39.domaincontrol.com
ns40.domaincontrol.com

You’ll need to create a resource record (server domain name = IP address), for instance:

zpm.example.com = 104.199.6.32

You do this by creating an A-type record in the zone example.com. After you save this record, other DNS servers all around the world will eventually know about this update and you’ll be able to refer to your zpm-registry by name: zpm.example.com:52773/registry/.

Domains and Kubernetes

As you might remember, the zpm service IP address was created during our Kubernetes Service (Load Balancer type) deployment. If you’ve played with zpm-registry, removed it, and then decided to deploy zpm-registry again, you’re likely to get another IP address. If that’s the case, you should again go to the DNS registrar web console and set a new IP address for zpm.example.com.
There’s another way to do this. During your Kubernetes deployment, you can deploy a helper tool, called External DNS, that can retrieve a newly created IP address from Kubernetes Services or from Ingress and create the corresponding DNS record. The tool doesn’t support all registrars, but it does support Google Cloud DNS, which can be responsible for DNS zones, providing name servers, and storing your resource records.

To use Google Cloud DNS, we need to transfer responsibility to it for our DNS zone example.com in the registrar web console. To do this, replace the name servers provided by domain registrar with the ones Google Cloud DNS provides. We will need to create the example.com zone in the Google console and copy/paste the name servers provided by Google. See details below.

Let’s see how to add External DNS and create our Google Cloud DNS in the code. However, to save space, I’ll include only parts of the code here. As I noted earlier, you’ll find the complete samples in GitHub GKE TLS repository.

Adding External DNS

To deploy this application to the GKE, we’ll use the power of Helm. You’ll find both the article An Introduction to Helm, the Package Manager for Kubernetes and the official documentation helpful.

Add these lines as the last job in the "kubernetes-deploy" stage in the pipeline file. Also add several new variables in the "env" section:


$ cat <root_repo_dir>/.github/workflows/workflow.yaml
...
env:
...
  DNS_ZONE: example.com
  HELM_VERSION: 3.1.1
  EXTERNAL_DNS_CHART_VERSION: 2.20.6
...
jobs:
...
  kubernetes-deploy:
...
  steps:
...
    - name: Install External DNS
      run: |
        wget -q https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz
        tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz
  cd linux-amd64 
        ./helm version
        ./helm repo add bitnami https://charts.bitnami.com/bitnami
        gcloud container clusters get-credentials ${GKE_CLUSTER} --zone ${GKE_ZONE} --project ${PROJECT_ID}
        echo ${GOOGLE_CREDENTIALS} > ./credentials.json
        kubectl create secret generic external-dns --from-file=./credentials.json --dry-run -o yaml | kubectl apply -f -
        ./helm upgrade external-dns bitnami/external-dns \
          --install \
          --atomic \
          --version=${EXTERNAL_DNS_CHART_VERSION} \
          --set provider=google \
          --set google.project=${PROJECT_ID} \
          --set google.serviceAccountSecret=external-dns \
          --set registry=txt \
          --set txtOwnerId=k8s \
          --set policy=sync \
          --set domainFilters={${DNS_ZONE}} \
          --set rbac.create=true

You can read more about parameters set by --set in the External DNS chart documentation. For now, just add these lines and let’s proceed.

Creating Cloud DNS

First, add a new file under the <root_repo_dir>/terraform/ directory. Remember to use your domain zone:

$ cat <root_repo_dir>/terraform/clouddns.tf
resource "google_dns_managed_zone" "my-zone" {
  name = "zpm-zone"
  dns_name = "example.com."
  description = "My DNS zone"
}

Also be sure that the user on whose behalf Terraform works has at least the following roles (note the DNS Administrator and Kubernetes Engine Admin roles):

If you run the GitHub Actions pipeline, these will be created.
With External DNS and Cloud DNS ready (though virtually at the moment), we can now consider how to expose our zpm-service in a way different from the load balancer service type.

Kubernetes

Our zpm-service is now exposed using a regular Kubernetes service load balancer. See the file <root_repo_dir>/k8s/service.yaml. You can read more about Kubernetes Services in Exposing applications using services. What’s important for us is that load balancer services don’t have the ability to set the domain name, and the actual Google Load Balancers created by those Kubernetes Services resources work on the TCP/UDP level of the OSI. This level knows nothing about HTTP and certificates. So we need to replace the Google network load balancer with an HTTP load balancer. We can create such a load balancer using the Kubernetes Ingress resource instead of the Kubernetes Service.

At this point, it would probably be a good idea for you to read Kubernetes NodePort vs LoadBalancer vs Ingress? When should I use what?.

Now let’s continue. What does Ingress usage mean in the code? Go to the <root_repo_dir>/k8s/ directory and make the following changes. The Service should be of type NodePort:


$ cat /k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: zpm-registry
  namespace: iris
spec:
  selector:
    app: zpm-registry
  ports:
  - protocol: TCP
    port: 52773
    targetPort: 52773
  type: NodePort

And you need to add the Ingress manifest:

$ cat <root_repo_dir>/k8s/ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: gce
    networking.gke.io/managed-certificates: zpm-registry-certificate
    external-dns.alpha.kubernetes.io/hostname: zpm.example.com
  name: zpm-registry
  namespace: iris
spec:
  rules:
  - host: zpm.example.com
    http:
      paths:
      - backend:
          serviceName: zpm-registry
          servicePort: 52773
        path: /*

Also, add the line marked in bold into the workflow file:


$ cat /.github/workflows/workflow.yaml
...
- name: Apply Kubernetes manifests
  working-directory: ./k8s/
...
    kubectl apply -f service.yaml
    kubectl apply -f ingress.yaml

If you deploy this manifest, the Google Cloud controller will create an external HTTP load balancer, which will listen for traffic to the host zpm.example.com and send this traffic to all Kubernetes nodes on the port that’s opened during Kubernetes NodePort Service deployment. This port is arbitrary, but you don’t need to worry about it — it’s completely automated.
We can define a more detailed configuration of Ingress using annotations:

annotations:
  kubernetes.io/ingress.class: gce
  networking.gke.io/managed-certificates: zpm-registry-certificate
  external-dns.alpha.kubernetes.io/hostname: zpm.example.com

The first line says we’re using the "gce" Ingress controller. This entity creates an HTTP load balancer according to settings in the Ingress resource.

The second line is the one that relates to certificates. We’ll return to this setting later.

The third line sets the external DNS that should bind the specified hostname (zpm.example.com) to the IP address of the HTTP load balancer.

If you did deploy this manifest, you’d see (after 10 minutes or so) that an Ingress has been created, but not everything else worked:

What are those "Backend services," one of which seems to be bad?

See their names, like "k8s-be-31407-..." and "k8s-be-31445-..."? These are ports opened on our single Kubernetes node. Your ports are likely to have different numbers.

31407 is a port that’s opened for health checks, which the HTTP load balancer constantly sends to be sure that node is alive.

You can take a look at those health check results by connecting to Kubernetes and proxying Node Port 31407:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID>
$ kubectl proxy &
$ curl localhost:8001/healthz
ok
$ fg
^Ctrl+C

Another port, 31445, is a NodePort opened for our zpm-registry service:


$ kubectl -n iris get svc
NAME          TYPE  CLUSTER-IP   EXTERNAL-IP PORT(S)      AGE
zpm-registry  NodePort 10.23.255.89       52773:31445/TCP 24m

And the default health checks the HTTP load balancer sends a report that this service is down. Is it really?

What are those Health checks in more detail? Click on the Backend service name. Scroll down a little to see the Health check name:

Click the Health check name to see details:

We should replace the Health check "/" path with a path understandable by IRIS, like "/csp/sys/UtilHome.csp":


$ curl -I localhost:52773/csp/sys/UtilHome.csp
Handling connection for 52773
HTTP/1.1 200 OK

So, how can we set that new path in the Health check. The answer with readiness/liveness probes as they’re used, if present, in place of the default "/" path.

So, let’s add those probes to the Kubernetes StatefulSet manifest:


$ cat /k8s/statefulset.tpl
...
containers:
- image: DOCKER_REPO_NAME:DOCKER_IMAGE_TAG
... 
  ports:
  - containerPort: 52773
    name: web
  readinessProbe:
    httpGet:
      path: /csp/sys/UtilHome.csp
      port: 52773
    initialDelaySeconds: 10
    periodSeconds: 10
  livenessProbe:
    httpGet:
      path: /csp/sys/UtilHome.csp
      port: 52773
    periodSeconds: 10
  volumeMounts:
  - mountPath: /opt/zpm/REGISTRY-DATA
    name: zpm-registry-volume
  - mountPath: /mount-helper
    name: mount-helper

So far, we’ve made several changes just to open a door for the main event: the certificate. Let’s get it and run all this finally.

Getting the Certificate

I highly recommended you watch these videos that describe different ways to obtain SSL/TLS certificates for Kubernetes:

In short, there are several ways to get a certificate: openssl self-signed, manual certbot by Let’s Encrypt, automated cert-manager connected to Let’s Encrypt and, finally, the native Google approach, Managed Certificates, which is our choice for its simplicity.

Let’s add it:

$ cat <root_repo-dir>/k8s/managed-certificate.yaml
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: zpm-registry-certificate
  namespace: iris
spec:
  domains:
  - zpm.example.com

Add the bolded line to the deployment pipeline:


$ cat /.github/workflows/workflow.yaml
...
- name: Apply Kubernetes manifests
  working-directory: ./k8s/
  run: |
    gcloud container clusters get-credentials ${GKE_CLUSTER} --zone ${GKE_ZONE} --project ${PROJECT_ID}
    kubectl apply -f namespace.yaml
    kubectl apply -f managed-certificate.yaml
    kubectl apply -f service.yaml
...

Enabling it in Ingress was done before as the Ingress annotation, remember?


$ cat /k8s/ingress.yaml
...
annotations:
  kubernetes.io/ingress.class: gce
  networking.gke.io/managed-certificates: zpm-registry-certificate
...

So now we’re ready to push all those changes to the repository:

$ git add .github/ terraform/ k8s/
$ git commit -m "Add TLS to GKE deploy"
$ git push

After 15 minutes or so (cluster provisioning should be done), you should see a "green" Ingress:

Then you should set at least two name servers provided by Google in your domain name registrars console (if you use another registrar than Google Domains):

We won’t describe this process as it’s specific to each registrar and usually well-described in the registrar’s documentation.

Google already knows about the new resource record created automatically by ExternalDNS:

$ dig +short @ns-cloud-b1.googledomains.com. zpm.example.com
34.102.202.2

But it will take some time until this record is propagated all around the world. Eventually, however, you’ll receive:

$ dig +short zpm.example.com
34.102.202.2

It’s a good idea to check the certificate status:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID>


$ kubectl -n iris get managedcertificate zpm-registry-certificate -ojson | jq '.status'
{
  "certificateName": "mcrt-158f20bb-cdd3-451d-8cb1-4a172244c14f",
  "certificateStatus": "Provisioning",
  "domainStatus": [
    {
      "domain": "zpm.example.com",
      "status": "Provisioning"
    }
  ]
}

You can read about the various status meanings on the Google-managed SSL certificate status page. Note that you might have to wait an hour or so for the initial certificate provisioning.

You might encounter the domainStatus "FailedNotVisible." If so, check that you’ve really added Google name servers in your DNS registrar console.

You should wait until both certificateStatus and domainStatus become available, and you may even have to wait a little longer, as stated in Google-manages SSL certificate status. Finally, however, you should be able to call zpm-registry with HTTPS:

curl -XGET -u _system:SYS https://zpm.example.com/registry/packages/-/all

Conclusion

Google does a great job creating all required Google resources based on Kubernetes resources.

In addition, Google Managed Certificates is a cool feature that greatly simplifies obtaining a certificate.

As always, don’t forget to remove Google resources (GKE, CloudDNS) when you no longer need them as they cost money.

Deploying InterSystems IRIS Solution into GCP Kubernetes Cluster GKE Using CircleCI

Mikhail Khomenko — Fri, 11 Sep 2020 08:07:40 +0000

Most of us are more or less familiar with Docker. Those who use it like it for the way it lets us easily deploy almost any application, play with it, break something and then restore the application with a simple restart of the Docker container.
InterSystems also likes Docker. The InterSystems OpenExchange project contains a number of examples that run InterSystems IRIS images in Docker containers that are easy to download and run. You’ll also find other useful components, such as the Visual Studio IRIS plugin.
It’s easy enough to run IRIS in Docker with additional code for specific use cases, but if you want to share your solutions with others, you’ll need some way to run commands and repeat them after each code update. In this article, we’ll see how to use Continuous Integration/Continuous Delivery (CI/CD) practices to simplify that process.

Setting Up

We’ll start with a simple REST API application based on IRIS. The details of the application can be found in the video Creating REST API with InterSystems IRIS, ObjectScript and Docker. Let’s see how we could share similar applications with others using CI/CD.

Initially, we’ll clone the code into a personal GitHub repository. If you don’t have an account on GitHub, sign up for one. For convenience, add access via SSH so you don’t need to enter a password with each pull or push. Then go to the intersystems-community/objectscript-rest-docker-template project page on GitHub and click the "Use this Template" button to create your own version of the repo based on the template. Give it a name like "my-objectscript-rest-docker-template".

Now pull the project to your local machine:

$ git clone git@github.com:<your_account>/my-objectscript-rest-docker-template.git

Next, we’ll add a REST endpoint in the spirit of "hello, world!".

Endpoints are defined in the src/cls/Sample/PersonREST.cls class. Our endpoint will look like this (defined before the first <Route>):

<Route Url="/helloworld" Method="GET" Call="HelloWorld"/>
<Route Url="/all" Method="GET" Call="GetAllPersons"/>
...

It calls the HelloWorld method:

ClassMethod HelloWorld() As %Status
{
    Write "Hello, world!"
    Quit $$$OK
}

Now we need to consider how this works when pushing to a remote repository. We need to:

Build a Docker image.
Save the Docker image.
Run the container based on this image.

We’ll use the CircleCI service, which is already integrated with GitHub, to build the Docker image. And we’ll use Google Cloud, which allows you to store Docker images and run containers based on them in Kubernetes. Let’s delve into this a little.

Google Cloud Prerequisites

Let’s assume you’ve registered for an account with Google Cloud, which provides a free tier of services. Create a project with the name "Development", then create a Kubernetes cluster by clicking the "Create cluster" button:

For the demo, select "Your first cluster" on the left. Choose a newer version of Kubernetes and a machine type of n1-standard-1. For our purposes, one machine should be enough.

Click the Create button, then set up a connection to the cluster. We’ll use the kubectl and gcloud utilities:

$ gcloud init
[2] Create a new configuration
Configuration name. "development"
[2] Log in with a new account
Pick cloud project to use
configure a default Compute Region and Zone? (Y/n)? y
Here europe-west-1b was chosen

$ gcloud container clusters get-credentials dev-cluster --zone europe-west1-b --project <project_id>

You can get the last command by clicking the "Connect" button:

Check the status from kubectl:

$ kubectl config current-context
gke_possible-symbol-254507_europe-west1-b_dev-cluster

$ kubectl get nodes
NAME                                     STATUS    ROLES   AGE     VERSION
gke-dev-cluster-pool-2-8096d93c-fw5w     Ready     <none>     17m     v1.14.7-gke.10

Now create a directory called k8s/ under the root project directory to hold the three files that describe the future application in Kubernetes: Namespace, which describes the workspace, Deployment, and Service:

$ cat namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: iris

$ cat deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: iris-rest
  namespace: iris
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: iris
  template:
    metadata:
      labels:
        app: iris
    spec:
      containers:
      - image: eu.gcr.io/iris-rest:v1
        name: iris-rest
        ports:
        - containerPort: 52773
          name: web

$ cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: iris-rest
  namespace: iris
spec:
  selector:
    app: iris
  ports:
  - protocol: TCP
    port: 52773
    targetPort: 52773
  type: LoadBalancer

Send those definitions from your k8s/ directory to the Google Kubernetes Engine (GKE):

$ kubectl apply -f namespace.yaml
$ kubectl apply -f deployment.yaml -f service.yaml

Things won’t be working correctly yet, since we haven’t yet sent the eu.gcr.io/iris-rest:v1 image to the Docker registry, so we see an error:

$ kubectl -n iris get po
NAME                            READY       STATUS    RESTARTS    AGE
iris-rest-64cdb48f78-5g9hb      0/1         ErrImagePull             0            50s


$ kubectl -n iris get svc
NAME          TYPE             CLUSTER-IP     EXTERNAL-IP     PORT(S)     AGE
iris-rest     LoadBalancer     10.0.13.219     <pending>     52773:31425/TCP     20s

When Kubernetes sees a LoadBalancer service, it tries to create a balancer in the Google Cloud environment. If it succeeds, the service will get a real IP address instead of External IP = <pending>.

Before leaving Kubernetes for a bit, let's give CircleCI the ability to push Docker images into the registry and restart Kubernetes deployments by creating a service account. Give your service account EDITOR permission to the project. You’ll find information here on creating and storing a service account.

A bit later, when we create and set up the project in CircleCI, you’ll need to add the following three environment variables:

The names of these variables speak for themselves. The value of GCLOUD_SERVICE_KEY is the JSON structure Google sends you when you press "Create key" and select a key in the JSON format after creating the Service Account:

CircleCI

Let's turn our attention to CircleCI now, where we’ll register using our GitHub account (click Sign Up, then Sign Up with GitHub). After registration, you’ll see the dashboard with projects from your GitHub repository listed on the Add Project tab. Click the Set Up Project button for "my-objectscript-rest-docker-template" or whatever you named the repository created from the objectscript-rest-docker-template repo:

Note: all CircleCI screenshots are made as of October 2019. Changes may occur in new versions.

The page that opens tells you how to make your project work with CircleCI. The first step is to create a folder called .circleci and add a file named config.yml to it. The structure of this configuration file is well described in the official documentation. Here are the basic steps the file will contain:

Pull the repository
Build the Docker image
Authenticate with Google Cloud
Upload image to Google Docker Registry
Run the container based on this image in GKE

With any luck, we’ll find some already created configurations (called orbs) we can use. There are certified orbs and third-party ones. The certified GCP-GKE orb had a number of limitations at the moment of initial writing, so let's take a third-party orb — duksis — that meets our needs. Using it, the configuration file turns into (replace names — for example, the cluster name — with correct ones for your implementation):

$ cat .circleci/config.yml
version: 2.1
orbs:
  gcp-gke: duksis/gcp-gke@0.1.9
workflows:
  main:
    jobs:
    - gcp-gke/publish-and-rollout-image:
         google-project-id: GOOGLE_PROJECT_ID
         gcloud-service-key: GCLOUD_SERVICE_KEY
         registry-url: eu.gcr.io
         image: iris-rest
         tag: ${CIRCLE_SHA1}
         cluster: dev-cluster
         namespace: iris
         deployment: iris-rest
         container: iris-rest

The initial configuration of the publish-and-rollout-image task can be viewed on the project page.

We don’t actually need the final three notification steps of this orb, which is good because they won’t work anyway without some additional variables. Ideally, you can prepare your own orb once and use it many times, but we won’t get into that now.

Note that the use of third-party orbs has to be specifically allowed on the "Organization settings" tab in CircleCI:

Finally, it’s time to send all our changes to GitHub and CircleCI:

$ git add .circleci/ k8s/ src/cls/Sample/PersonREST.cls
$ git commit -m "Deploy project to GKE using CircleCI"
$ git push

Let’s check the CircleCI dashboard:

If you forgot to add Google Service Account keys, here’s what you’ll soon see:

So don’t forget to add those environment variables as described in the end of the Google Cloud Prerequisites section. If you forgot, update that information, then click "Rerun workflow".

If the build is successful you’ll see a green bar:

You can also check the Kubernetes pod state separately from the CircleCI Web UI:

$ kubectl -n iris get po -w
NAME                            READY     STATUS     RESTARTS     AGE
iris-rest-64chdb48f78-q5sbw     0/1     ImagePullBackOff     0     15m
…
iris-rest-5c9c86c768-vt7c9     1/1     Running     0     23s

That last line — 1/1 Running — is a good sign.

Let’s test it. Remember, your IP address will differ from mine. Also, you’ll have to figure out about passwords over HTTP yourself as it’s out of scope for this article.

$ kubectl -n iris get svc
NAME          TYPE             CLUSTER-IP     EXTERNAL-IP     PORT(S)     AGE
iris-rest     LoadBalancer     10.0.4.242     23.251.143.124     52773:30948/TCP     18m


$ curl -XGET -u _system:SYS 23.251.143.124:52773/person/helloworld
Hello, world!

$ curl -XPOST -H "Content-Type: application/json" -u _system:SYS 23.251.143.124:52773/person/ -d '{"Name":"John Dou"}'

$ curl -XGET -u _system:SYS 23.251.143.124:52773/person/all
[{"Name":"John Dou"},]

It seems the application works. You can continue with tests described on project page.

In sum, the combination of using GitHub, CircleCI, and Google Cloud Kubernetes Engine looks quite promising for testing and deployment of IRIS applications, even though it’s not completely free. Also, do not forget that running Kubernetes cluster can gradually eat your virtual (and then real) money. We are not responsible for any charges you may incur.

Deploying a Simple IRIS-Based Web Application Using Amazon EKS

Mikhail Khomenko — Tue, 04 Aug 2020 07:12:42 +0000

We’re going to deploy a simple IRIS application to Amazon Web Services using its Elastic Kubernetes Service (EKS).

Please, fork the IRIS project to your own private repository. It’s called <username>/my-objectscript-rest-docker-template in this article. <root_repo_dir> is its root directory.

Before getting started, install the AWS command-line tool and, for Kubernetes cluster creation, eksctl, a simple CLI utility. If you choose aws2, you’ll need to set aws2 usage in kube config file as described here.

AWS EKS

Like AWS resources in general, EKS is not free. But you can create a free-tier account to play with AWS features. Keep in mind, though, that not everything you want to play with is included in the free tier. So, to manage your current budget and understand the financial issues, read What is AWS Billing and Cost Management? and Control your AWS costs.

We’ll assume you already have an AWS account and root access to it, and that you don’t use this root access but have created a user with admin permissions. You’ll need to put the access key and secret key of this user into the AWS credentials file under the [dev] profile (or whatever you choose to name the profile):

$ cat ~/.aws/credentials
[dev]
aws_access_key_id = ABCDEFGHIJKLMNOPQRST
aws_secret_access_key = 1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ1234

We’re going to create resources in the AWS "eu-west-1" region, but you should choose the region closest to your location and replace “eu-west-1” by your region everywhere below in the text.

By the way, all needed files (.circleci/, eks/, k8s/) are also stored here to simplify copying and pasting.

All required EKS resources will be created from scratch. You’ll find the Amazon EKS Workshop to be a good resource to get an initial impression.

Now let’s check our access to AWS (we’ve used a dummy account here):

$ export AWS_PROFILE=dev

$ aws sts get-caller-identity
{
  "Account": "012345678910",
  "UserId": " ABCDEFGHIJKLMNOPQRSTU",
  "Arn": "arn:aws:iam::012345678910:user/FirstName.LastName"
}

$ eksctl version
[ℹ] version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.10.2"}

We could run "eksctl create cluster --region eu-west-1" now, relying on the fact that all the default settings are good for us, or we can manage our own settings by creating a configuration file and using it.

The latter is preferable because it allows you to store such a file in a version control system (VCS). Examples of configurations can be found here. After reading about the different settings here, let’s try to create our own configuration):

$ mkdir <root_repo_dir>/eks; cd <root_repo_dir>/eks

$ cat cluster.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: dev-cluster
  region: eu-west-1
  version: '1.14'

vpc:
  cidr: 10.42.0.0/16
  nat:
    gateway: Single
  clusterEndpoints:
    publicAccess: true
    privateAccess: false

nodeGroups:
  - name: ng-1
    amiFamily: AmazonLinux2
    ami: ami-059c6874350e63ca9  # AMI is specific for a region
    instanceType: t2.medium
    desiredCapacity: 1
    minSize: 1
    maxSize: 1

    # Worker nodes won't have an access FROM the Internet
    # But have an access TO the Internet through NAT-gateway
    privateNetworking: true

    # We don't need to SSH to nodes for demo
    ssh:
      allow: false

    # Labels are Kubernetes labels, shown when 'kubectl get nodes --show-labels'
    labels:
      role: eks-demo
    # Tags are AWS tags, shown in 'Tags' tab on AWS console'
    tags:
      role: eks-demo

# CloudWatch logging is disabled by default to save money
# Mentioned here just to show a way to manage it
#cloudWatch: 
#  clusterLogging:
#    enableTypes: []

Note that "nodeGroups.desiredCapacity = 1" would make no sense in a production environment, but it’s fine for our demo.
Also note that AMI images could differ between regions. Look for "amazon-eks-node-1.14" and choose one of the latest:

Now let’s create the cluster - the control plane and worker nodes:

$ eksctl create cluster -f cluster.yaml

By the way, when you no longer need a cluster, you can use the following to delete it:

$ eksctl delete cluster --name dev-cluster --region eu-west-1 --wait

Creating a cluster takes about 15 minutes. During this time you can look at the eksctl output:

You can also view the CloudFormation console, which will have two stacks. You can drill down into each one and look at the Resources tab to see exactly what will be created, and at the Events tab to check the current state of the resources creation.

The cluster was successfully created, although you can see in the eksctl output that we had difficulties connecting to it: "unable to use kubectl with the EKS cluster".
Let's fix this by installing the aws-iam-authenticator and using it to create a kube context:

$ which aws-iam-authenticator
/usr/local/bin/aws-iam-authenticator

$ aws eks update-kubeconfig --name dev-cluster --region eu-west-1

$ kubectl get nodes
NAME                                       STATUS  ROLES   AGE   VERSION                                                                              
ip-10-42-140-98.eu-west-1.compute.internal Ready   <none>   1m    v1.14.7-eks-1861c5

It should work now, but we created a cluster with a user who has administrator rights. For the regular deployment process from CircleCI, it’s better to create a special AWS user, named, in this case, CircleCI, with only programmatic access and the following policies attached:

The first policy is built into AWS, so you only need to select it. The second one should be created on your own. Here is a creation process description. The policy "AmazonEKSDescribePolicy" should look like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}

After user creation, save the user’s access key and secret access key — we’ll need them soon.

We also want to give this user rights within the Kubernetes cluster itself, as described in How do I provide access to other users and roles after cluster creation in Amazon EKS?. In short, after creating the EKS cluster, only the IAM user, creator, has access to it. To add our CircleCI user, we need to replace default empty "mapUsers" section in the cluster AWS authentication settings (configmap aws-auth, 'data' section) by the following lines using kubectl edit (use your own Account Id instead of ‘01234567890’):

$ kubectl -n kube-system edit configmap aws-auth
...
data:
...
  mapUsers: |
    - userarn: arn:aws:iam::01234567890:user/CircleCI
      username: circle-ci
      groups:
        - system:masters

We’ll use the Kubernetes manifests from this article (see the "Google Cloud Prerequisite" section) with one change: in the deployment image field we use placeholders. We’ll store those manifests in the <root_repo_dir>/k8s directory. Note that the deployment file was renamed to deployment.tpl:

$ cat <root_repo_dir>/k8s/deployment.tpl
...
spec:
containers:
- image: DOCKER_REPO_NAME/iris-rest:DOCKER_IMAGE_TAG
...

CircleCI

The deployment process on the CircleCI side is similar to the process used for GKE:

Pull the repository
Build the Docker image
Authenticate with Amazon Cloud
Upload the image to Amazon Elastic Container Registry (ECR)
Run the container based on this image in AWS EKS

We’ll take advantage of already created and tested CircleCI configuration templates: orbs.

aws-ecr orb for building an image and pushing it to ECR
aws-eks orb for AWS authentication
kubernetes orb for Kubernetes manifests deployment

Our deployment configuration looks like this:

$ cat <root_repo_dir>/.circleci/config.yml
version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@6.5.0
  aws-eks: circleci/aws-eks@0.2.6
  kubernetes: circleci/kubernetes@0.10.1

jobs:
  deploy-application:
    executor: aws-eks/python3
    parameters:
      cluster-name:
        description: |
          Name of the EKS cluster
        type: string
      aws-region:
        description: |
          AWS region
        type: string
      account-url:
        description: |
          Docker AWS ECR repository url
        type: string
      tag:
        description: |
          Docker image tag
        type: string
    steps:
      - checkout
      - run:
          name: Replace placeholders with values in deployment template
          command: |
            cat k8s/deployment.tpl |\
            sed "s|DOCKER_REPO_NAME|<< parameters.account-url >>|" |\
            sed "s|DOCKER_IMAGE_TAG|<< parameters.tag >>|" > k8s/deployment.yaml; \
            cat k8s/deployment.yaml
      - aws-eks/update-kubeconfig-with-authenticator:
          cluster-name: << parameters.cluster-name >>
          install-kubectl: true
          aws-region: << parameters.aws-region >>
      - kubernetes/create-or-update-resource:
          action-type: apply
          resource-file-path: "k8s/namespace.yaml"
          show-kubectl-command: true
      - kubernetes/create-or-update-resource:
          action-type: apply
          resource-file-path: "k8s/deployment.yaml"
          show-kubectl-command: true
          get-rollout-status: true
          resource-name: deployment/iris-rest
          namespace: iris
      - kubernetes/create-or-update-resource:
          action-type: apply
          resource-file-path: "k8s/service.yaml"
          show-kubectl-command: true
          namespace: iris
workflows:
  main:
    jobs:
    - aws-ecr/build-and-push-image:
        aws-access-key-id: AWS_ACCESS_KEY_ID
        aws-secret-access-key: AWS_SECRET_ACCESS_KEY
        region: AWS_REGION
        account-url: AWS_ECR_ACCOUNT_URL
        repo: iris-rest
        create-repo: true
        dockerfile: Dockerfile-zpm
        path: .
        tag: ${CIRCLE_SHA1}
    - deploy-application:
        cluster-name: dev-cluster
        aws-region: eu-west-1
        account-url: ${AWS_ECR_ACCOUNT_URL}
        tag: ${CIRCLE_SHA1}
        requires:
          - aws-ecr/build-and-push-image

The Workflows section contains a list of jobs, each of which can be either called from an orb, such as aws-ecr/build-and-push-image, or defined directly in the configuration using "deploy-application".

The following code means that the deploy-application job will be called only after the aws-ecr/build-and-push-image job finishes:

requires:
- aws-ecr/build-and-push-image

The Jobs section contains a description of the deploy-application job, with a list of steps defined, including:

checkout, to pull from a Git repository
run, to run a script that dynamically sets the Docker-image repository and tag
aws-eks/update-kubeconfig-with-authenticator, which uses aws-iam-authenticator to set up a connection to Kubernetes
kubernetes/create-or-update-resource, which is used several times as a way to run "kubectl apply" from CircleCI

We use variables and they, of course, should be defined in CircleCI on the "Environment variables" tab:

The following table shows the meaning of the variables used:

Variable	Meaning
AWS_ACCESS_KEY_ID	Access key of CircleCI IAM user
AWS_SECRET_ACCESS_KEY	Secret key of CircleCI IAM user
AWS_REGION	eu-west-1, in this case
AWS_ECR_ACCOUNT_URL	URL of the AWS ECR Docker Registry, such as 01234567890.dkr.ecr.eu-west-1.amazonaws.com, where ‘01234567890’ is the account ID

Here’s how we trigger the deployment process:

$ git add .circleci/ eks/ k8s/
$ git commit -m “AWS EKS deployment”
$ git push

This will show the two jobs in this workflow:

Both jobs are clickable, and this allows you to see details of the steps taken.
Deployment takes several minutes. Once it completes, we can check the status of the Kubernetes resources and of the IRIS application itself:

$ kubectl -n iris get pods -w    # Ctrl+C to stop

$ kubectl -n iris get service    
NAME       TYPE        CLUSTER-IP      EXTERNAL-IP              PORT(S)      AGE                                                                                                                                          
iris-rest LoadBalancer 172.20.190.211  a3de52988147a11eaaaff02ca6b647c2-663499201.eu-west-1.elb.amazonaws.com  52773:32573/TCP 15s

Allow several minutes to propagate the DNS-record. Until then you’ll receive a "Could not resolve host" error when running curl:

$ curl -XPOST -H "Content-Type: application/json" -u _system:SYS a3de52988147a11eaaaff02ca6b647c2-663499201.eu-west-1.elb.amazonaws.com:52773/person/ -d '{"Name":"John Dou"}'

$ curl -XGET -u _system:SYS a3de52988147a11eaaaff02ca6b647c2-663499201.eu-west-1.elb.amazonaws.com:52773/person/all
[{"Name":"John Dou"},]

Wrapping up

At first glance, deployment to AWS EKS looks more complex than to GKE, but it’s not really much different. If your organization uses AWS, you now know how to add Kubernetes to your stack.

Note that the EKS API was extended to support managed groups. These allow you to deploy the control plane and the data plane as a whole, and they look promising. Moreover, Fargate, the AWS serverless compute engine for containers, is now available.

Finally, a quick note about AWS ECR: don’t forget to set up a lifecycle policy for your images.

Automating GKE creation on CircleCI builds

Mikhail Khomenko — Fri, 31 Jul 2020 07:34:28 +0000

Creating GKE cluster manually (or through gcloud) is easy, but the modern Infrastructure-as-Code (IaC) approach advises that the description of the Kubernetes cluster should be stored in the repository as code as well. How to write this code is determined by the tool that’s used for IaC.

In the case of Google Cloud, there are several options, among them Deployment Manager and Terraform. Opinions are divided as to which is better: if you want to learn more, read this Reddit thread Opinions on Terraform vs. Deployment Manager? and the Medium article Comparing GCP Deployment Manager and Terraform.

For this article we’ll choose Terraform, since it’s less tied to a specific vendor and you can use your IaC with different cloud providers.

We’ll assume you already have a Google account, and that you’ve created a project named, for instance, "Development". In this article, its ID is shown as <PROJECT_ID>. In the examples below, change it to the ID of your own project.

Keep in mind that Google isn’t free, although it has a free tier. Be sure to control your expenses.

You should fork the original repository. We’ll call this fork “my-objectscript-rest-docker-template” and refer to its root directory as "<root_repo_dir>" throughout this article.

All code samples are stored in this repo to simplify copying and pasting.

The following diagram depicts the whole deployment process in one picture:

So, let's install the latest version of Terraform at the time of initial writing:

$ terraform version
Terraform v0.12.17

The version is important here, because many examples on the Internet use earlier versions, and 0.12 brought many changes. Update: 0.13 brought yet more changes.

We want Terraform to perform certain actions (use certain APIs) in our GCP account. To enable this, create a Service Account with the name 'terraform', and enable the Kubernetes Engine API. Don’t worry about how we’re going to achieve this — just read further and your questions will be addressed.

Let's try an example with the gcloud utility, although we could also use the Web Console.

We're going to use a couple different commands in the following examples. See the following documentation topics for more details on these commands and features:

Now let's walk through the example.

$ gcloud init

We won’t discuss all of the setup details here. You can read a little more in this article. For this example, run the following commands:

$ cd <root_repo_dir>
$ mkdir terraform; cd terraform
$ gcloud iam service-accounts create terraform --description "Terraform" --display-name "terraform"

Now let's add a few roles to the terraform service account besides “Kubernetes Engine Admin” (container.admin). These roles will be useful to us in the future.

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/container.admin

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/iam.serviceAccountUser

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/compute.viewer

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \
  --role roles/storage.admin

$ gcloud iam service-accounts keys create account.json \
  --iam-account terraform@<PROJECT_ID>.iam.gserviceaccount.com

Note that the last entry creates your account.json file. Be sure to keep this file secret.

$ gcloud projects list
$ gcloud config set project <PROJECT_ID>
$ gcloud services list --available | grep 'Kubernetes Engine'
$ gcloud services enable container.googleapis.com
$ gcloud services list --enabled | grep 'Kubernetes Engine'
container.googleapis.com Kubernetes Engine API

Next, let’s describe the GKE cluster in Terraform’s HCL language. Note that we use several placeholders here; replace them with your values:

Placeholder	Meaning	Example
<PROJECT_ID>	GCP project ID	possible-symbol-254507
<BUCKET_NAME>	Storage for Terraform state - should be unique	circleci-gke-terraform-demo
<REGION>	Region where resources will be created	europe-west1
<LOCATION>	Zone where resources will be created	europe-west1-b
<CLUSTER_NAME>	GKE cluster name	dev-cluster
<NODES_POOL_NAME>	GKE worker nodes pool name	dev-cluster-node-pool

Here’s the HCL configuration for the cluster in practice:

$ cat main.tf
terraform {
  required_version = "~> 0.12"
  backend "gcs" {
    bucket = "<BUCKET_NAME>"
    prefix = "terraform/state"
    credentials = "account.json"
  }
}

provider "google" {
  credentials = file("account.json")
  project = "<PROJECT_ID>"
  region = "<REGION>"
}

resource "google_container_cluster" "gke-cluster" {
  name = "<CLUSTER_NAME>"
  location = "<LOCATION>"
  remove_default_node_pool = true
  # In regional cluster (location is region, not zone) 
  # this is a number of nodes per zone 
  initial_node_count = 1
}

resource "google_container_node_pool" "preemptible_node_pool" {
  name = "<NODES_POOL_NAME>"
  location = "<LOCATION>"
  cluster = google_container_cluster.gke-cluster.name
  # In regional cluster (location is region, not zone) 
  # this is a number of nodes per zone
  node_count = 1

  node_config {
    preemptible = true
    machine_type = "n1-standard-1"
    oauth_scopes = [
      "storage-ro",
      "logging-write",
      "monitoring"
    ]
  }
}

To make sure the HCL code is in the proper format, Terraform provides a handy formatting command you can use:

$ terraform fmt

The code snippet shown above indicates that the created resources will be provided by Google, and the resources themselves are google_container_cluster and google_container_node_pool, which we designate preemptible for costs savings. We also choose to create our own pool instead of using the default.

Let’s focus briefly on the following setting:

terraform {
  required_version = "~> 0.12"
  backend "gcs" {
    Bucket = "<BUCKET_NAME>"
    Prefix = "terraform/state"
    credentials = "account.json"
  }
}

Terraform writes everything it's done into the status file and then uses this file for other work. For convenient sharing, it’s better to store this file somewhere in a remote place. A typical place is a Google Bucket.

Let's create this bucket. Use the name of your bucket instead of the placeholder <BUCKET_NAME>. Before bucket creation let’s check if <BUCKET_NAME> is available as it has to be unique across all GCP:

$ gsutil acl get gs://<BUCKET_NAME>

Good answer:

BucketNotFoundException: 404 gs://<BUCKET_NAME> bucket does not exist

"Busy" answer means you have to choose another name:

AccessDeniedException: 403 <YOUR_ACCOUNT> does not have storage.buckets.get access to <BUCKET_NAME>

Let’s also enable versioning, as Terraform recommends.

$ gsutil mb -l EU gs://<BUCKET_NAME>

$ gsutil versioning get gs://<BUCKET_NAME>
gs://<BUCKET_NAME>: Suspended

$ gsutil versioning set on gs://<BUCKET_NAME>

$ gsutil versioning get gs://<BUCKET_NAME>
gs://<BUCKET_NAME>: Enabled

Terraform is modular and needs to add a Google provider plugin to create something in GCP. We use the following command to do this:

$ terraform init

Let's look at what Terraform is going to do to create a GKE cluster:

$ terraform plan -out dev-cluster.plan

The command output includes details of the plan. If you have no objections, let's implement this plan:

$ terraform apply dev-cluster.plan

By the way, to delete the resources created by Terraform, run this command from the <root_repo_dir>/terraform/ directory:

$ terraform destroy -auto-approve

Let’s leave the cluster as is for a while and move on. But first note that we don’t want to push everything into the repository, so we’ll add several files to the exceptions:

$ cat <root_repo_dir>/.gitignore
.DS_Store
terraform/.terraform/
terraform/*.plan
terraform/*.json

Using Helm

As described in this article, we could store Kubernetes manifests as yaml files in the <root_repo_dir>/k8s/ directory, which we then sent to the cluster using the "kubectl apply" command.

This time we'll try a different approach: using the Kubernetes package manager Helm, which currently has version 3. Please, use version 3 or later because version 2 had Kubernetes-side security issues (see Running Helm in production: Security best practices for details). First, we’ll pack the Kubernetes manifests from our k8s/ directory into a Helm package, which is known as a chart. A Helm chart installed in Kubernetes is called a release. In a minimal configuration, a chart will consist of several files:

$ mkdir <root_repo_dir>/helm; cd <root_repo_dir>/helm
$ tree <root_repo_dir>/helm/
helm/
├── Chart.yaml
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   └── service.yaml
└── values.yaml

Their purpose is well-described on the official site. The best practices for creating your own charts are described in the The Chart Best Practices Guide in the Helm documentation.

Here’s what the contents of our files look like:

$ cat Chart.yaml
apiVersion: v2
name: iris-rest
version: 0.1.0
appVersion: 1.0.3
description: Helm for ObjectScript-REST-Docker-template application
sources:
- https://github.com/intersystems-community/objectscript-rest-docker-template
- https://github.com/intersystems-community/gke-terraform-circleci-objectscript-rest-docker-template

$ cat templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ template "iris-rest.name" . }}
  labels:
    app: {{ template "iris-rest.name" . }}
    chart: {{ template "iris-rest.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  strategy:
    {{- .Values.strategy | nindent 4 }}
  selector:
    matchLabels:
      app: {{ template "iris-rest.name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "iris-rest.name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
      - image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        name: {{ template "iris-rest.name" . }}
        ports:
        - containerPort: {{ .Values.webPort.value }}
          name: {{ .Values.webPort.name }}

$ cat templates/service.yaml
{{- if .Values.service.enabled }}
apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.service.name }}
  labels:
    app: {{ template "iris-rest.name" . }}
    chart: {{ template "iris-rest.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  selector:
    app: {{ template "iris-rest.name" . }}
    release: {{ .Release.Name }}
  ports:
  {{- range $key, $value := .Values.service.ports }}
    - name: {{ $key }}
{{ toYaml $value | indent 6 }}
  {{- end }}
  type: {{ .Values.service.type }}
  {{- if ne .Values.service.loadBalancerIP "" }}
  loadBalancerIP: {{ .Values.service.loadBalancerIP }}
  {{- end }}
{{- end }}

$ cat templates/_helpers.tpl
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}

{{- define "iris-rest.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "iris-rest.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}

$ cat values.yaml
namespaceOverride: iris-rest

replicaCount: 1

strategy: |
  type: Recreate

image:
  repository: eu.gcr.io/iris-rest
  tag: v1

webPort:
  name: web
  value: 52773

service:
  enabled: true
  name: iris-rest
  type: LoadBalancer
  loadBalancerIP: ""
  ports:
    web:
      port: 52773
      targetPort: 52773
      protocol: TCP

To create the Helm charts, install the Helm client and the kubectl command-line utility.

$ helm version
version.BuildInfo{Version:"v3.0.1", GitCommit:"7c22ef9ce89e0ebeb7125ba2ebf7d421f3e82ffa", GitTreeState:"clean", GoVersion:"go1.13.4"}

Create a namespace called "iris". It would be nice if this was created during the deployment, but initially it was not the case. Update: looks like, --create-namespace flag is currently supported.

First, add credentials for the cluster created by Terraform to kube-config:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID>
$ kubectl create ns iris

Confirm (without kicking off a real deploy) that Helm is going to create the following in Kubernetes:

$ cd <root_repo_dir>/helm
$ helm upgrade iris-rest \
  --install \
  . \
  --namespace iris \
  --debug \
  --dry-run

The output - the Kubernetes manifests - has been omitted for space here. If everything looks good, let’s deploy:

$ helm upgrade iris-rest --install . --namespace iris

$ helm list -n iris --all
iris-rest  iris  1  2019-12-14 15:24:19.292227564  +0200  EET  deployed    iris-rest-0.1.0  1.0.3

We see that Helm has deployed our application, but since we haven’t created the Docker image eu.gcr.io/iris-rest:v1 yet, Kubernetes can’t pull it (ImagePullBackOff):

$ kubectl -n iris get po
NAME                       READY  STATUS            RESTARTS AGE
iris-rest-59b748c577-6cnrt 0/1    ImagePullBackOff  0         10m

Let’s finish with it for now:

$ helm delete iris-rest -n iris

The CircleCI Side

Now that we’ve tried out Terraform and the Helm client, let’s put them to use during the deployment process on the CircleCI side.

$ cat <root_repo_dir>/.circleci/config.yml
version: 2.1

orbs:
  gcp-gcr: circleci/gcp-gcr@0.6.1

jobs:
  terraform:
    docker:
    # Terraform image version should be the same as when
    # you run terraform before from the local machine
      - image: hashicorp/terraform:0.12.17
    steps:
      - checkout
      - run:
          name: Create Service Account key file from environment variable
          working_directory: terraform
          command: echo ${TF_SERVICE_ACCOUNT_KEY} > account.json
      - run:
          name: Show Terraform version
          command: terraform version
      - run:
          name: Download required Terraform plugins
          working_directory: terraform
          command: terraform init
      - run:
          name: Validate Terraform configuration
          working_directory: terraform
          command: terraform validate
      - run:
          name: Create Terraform plan
          working_directory: terraform
          command: terraform plan -out /tmp/tf.plan
      - run:
          name: Run Terraform plan
          working_directory: terraform
          command: terraform apply /tmp/tf.plan
  k8s_deploy:
    docker:
      - image: kiwigrid/gcloud-kubectl-helm:3.0.1-272.0.0-218
    steps:
      - checkout
      - run:
          name: Authorize gcloud on GKE
          working_directory: helm
          command: |
            echo ${GCLOUD_SERVICE_KEY} > gcloud-service-key.json
            gcloud auth activate-service-account --key-file=gcloud-service-key.json
            gcloud container clusters get-credentials ${GKE_CLUSTER_NAME} --zone ${GOOGLE_COMPUTE_ZONE} --project ${GOOGLE_PROJECT_ID}
      - run:
          name: Wait a little until k8s worker nodes up
          command: sleep 30 # It’s a place for improvement
      - run:
          name: Create IRIS namespace if it doesn't exist
          command: kubectl get ns iris || kubectl create ns iris
      - run:
          name: Run Helm release deployment
          working_directory: helm
          command: |
            helm upgrade iris-rest \
              --install \
              . \
              --namespace iris \
              --wait \
              --timeout 300s \
              --atomic \
              --set image.repository=eu.gcr.io/${GOOGLE_PROJECT_ID}/iris-rest \
              --set image.tag=${CIRCLE_SHA1}
      - run:
          name: Check Helm release status
          command: helm list --all-namespaces --all
      - run:
          name: Check Kubernetes resources status
          command: |
            kubectl -n iris get pods
            echo
            kubectl -n iris get services
workflows:
  main:
    jobs:
      - terraform
      - gcp-gcr/build-and-push-image:
          dockerfile: Dockerfile
          gcloud-service-key: GCLOUD_SERVICE_KEY
          google-compute-zone: GOOGLE_COMPUTE_ZONE
          google-project-id: GOOGLE_PROJECT_ID
          registry-url: eu.gcr.io
          image: iris-rest
          path: .
          tag: ${CIRCLE_SHA1}
      - k8s_deploy:
          requires:
            - terraform
            - gcp-gcr/build-and-push-image

You’ll need to add several environment variables to your project on CircleCI side:

The GCLOUD_SERVICE_KEY is the CircleCI service account key, and TF_SERVICE_ACCOUNT_KEY is the Terraform service account key. Recall that the service account key is the whole content of account.json file.

Next, let’s push our changes to a repository:

$ cd <root_repo_dir>
$ git add .circleci/ helm/ terraform/ .gitignore
$ git commit -m "Add Terraform and Helm"
$ git push

The CircleCI UI dashboard should show that everything is ok:

Terraform is an idempotent tool and if the GKE cluster is present, the "terraform" job won’t do anything. If the cluster doesn’t exist, it will be created before Kubernetes deployment.
Finally, let’s check IRIS availability:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID>

$ kubectl -n iris get svc
NAME      TYPE          CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE   
iris-rest LoadBalancer  10.23.249.42  34.76.130.11  52773:31603/TCP   53s

$ curl -XPOST -H "Content-Type: application/json" -u _system:SYS 34.76.130.11:52773/person/ -d '{"Name":"John Dou"}'

$ curl -XGET -u _system:SYS 34.76.130.11:52773/person/all
[{"Name":"John Dou"},]

Conclusion

Terraform and Helm are standard DevOps tools and should be fine integrated with IRIS deployment.

They do require some learning, but after some practice, they can really save you time and effort.

Deploying InterSystems IRIS solution on GKE Using GitHub Actions

Mikhail Khomenko — Fri, 24 Jul 2020 07:25:12 +0000

In this article, we took a look at the CircleCI deployment system, which integrates perfectly with GitHub. Why then would we want to look any further? Well, GitHub has its own CI/CD platform called GitHub Actions, which is worth exploring. With GitHub Actions, you don’t need to rely on some external, albeit cool, service.

In this article we’re going to try using GitHub Actions to deploy the server part of InterSystems Package Manager, ZPM-registry, on Google Kubernetes Engine (GKE).

As with all systems, the build/deploy process essentially comes down to “do this, go there, do that,” and so on. With GitHub Actions, each such action is a job that consists of one or more steps, together known as a workflow. GitHub will search for a description of the workflow in the YAML file (any filename ending in .yml or .yaml) in your .github/workflows directory. See Core concepts for GitHub Actions for more details.

All further actions will be performed in the fork of the ZPM-registry repository. We’ll call this fork "zpm-registry" and refer to its root directory as "<root_repo_dir>" throughout this article. To learn more about the ZPM application itself see Introducing InterSystems ObjectScript Package Manager and The Anatomy of ZPM Module: Packaging Your InterSystems Solution.

All code samples are stored in this repository to simplify copying and pasting. The prerequisites are the same as in the article Automating GKE creation on CircleCI builds.

We’ll assume you’ve read that article and already have a Google account, and that you’ve created a project named "Development," as in the previous article. In this article, its ID is shown as <PROJECT_ID>. In the examples below, change it to the ID of your own project.

Keep in mind that Google isn’t free, although it has a free tier. Be sure to control your expenses.

Workflow Basics

Let’s get started.

A simple and useless workflow file might look like this:

$ cd <root_repo_dir>
$ mkdir -p .github/workflows
$ cat <root_repo_dir>/.github/workflows/workflow.yaml          
name: Traditional Hello World
on: [push]
jobs:
  courtesy:
    name: Greeting
    runs-on: ubuntu-latest
    steps:
    - name: Hello world
      run: echo "Hello, world!

When pushing to the repository, you need to execute a job named "Greeting," which consists of a single step: printing a welcome phrase. The job should run on a GitHub-hosted virtual machine called the Runner, with the latest version of Ubuntu installed.
After pushing this file to the repository, you should see on the Code GitHub tab that everything went well:

If the job had failed, you’d see a red X instead of a green checkmark. To see more, click on the green checkmark and then on Details. Or you can immediately go to the Actions tab:

You can learn all about the workflow syntax in the help document Workflow syntax for GitHub Actions.

If your repository contains a Dockerfile for the image build, you could replace the "Hello world" step with something more useful like this example from starter-workflows:


steps:
- uses: actions/checkout@v2
- name: Build the Docker image
  run: docker build . --file Dockerfile --tag my-image:$(date +%s)

Notice that a new step, "uses: action/checkout@v2", was added here. Judging by the name "checkout", it clones the repository, but where to find out more?

As in the case of CircleCI, many useful steps don’t need to be rewritten. Instead, you can take them from the shared resource called Marketplace. Look there for the desired action, and note that it’s better to take those that are marked as "By actions" (when you hover over - "Creator verified by Github").

The "uses" clause in the workflow reflects our intention to use a ready-made module, rather than writing one ourselves.

The implementations of the actions themselves can be written in almost any language, but JavaScript is preferred. If your action is written in JavaScript (or TypeScript), it will be executed directly on the Runner machine. For other implementations, the Docker container you specify will run with the desired environment inside, which is obviously somewhat slower. You can read more about actions in the aptly titled article, About actions.

The checkout action is written in TypeScript. And in our example, Terraform action is a regular bash script launched in Docker Alpine.

There’s a Dockerfile in our cloned repository, so let's try to apply our new knowledge. We’ll build the image of the ZPM registry and push it into the Google Container Registry. In parallel, we’ll create the Kubernetes cluster in which this image will run, and we’ll use Kubernetes manifests to do this.

Here’s what our plan, in a language that GitHub understands, will look like (but keep in mind that this is a bird's eye view with many lines omitted for simplification, so don’t actually use this config):

name: Workflow description
# Trigger condition. In this case, only on push to ‘master’ branch
on:
  push:
    branches:
    - master
# Here we describe environment variables available 
# for all further jobs and their steps
# These variables can be initialized on GitHub Secrets page
# We add “${{ secrets }}” to refer them
env:
  PROJECT_ID: ${{ secrets.PROJECT_ID }}

# Define a jobs list. Jobs/steps names could be random but
# it’s better to have they meaningful
jobs:
  gcloud-setup-and-build-and-publish-to-GCR:
    name: Setup gcloud utility, Build ZPM image and Publish it to Container Registry
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
    - name: Setup gcloud cli
    - name: Configure docker to use the gcloud as a credential helper
    - name: Build ZPM image
    - name: Publish ZPM image to Google Container Registry

  gke-provisioner:
    name: Provision GKE cluster
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
    - name: Terraform init
    - name: Terraform validate
    - name: Terraform plan
    - name: Terraform apply

  kubernetes-deploy:
    name: Deploy Kubernetes manifests to GKE cluster
    needs:
    - gcloud-setup-and-build-and-publish-to-GCR
    - gke-provisioner
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
    - name: Replace placeholders with values in statefulset template
    - name: Setup gcloud cli
    - name: Apply Kubernetes manifests

This is the skeleton of the working config in which there are no muscles, the real actions for each step. Actions can be accomplished with a simple console command ("run" or "run |" if there are several commands):

- name: Configure docker to use gcloud as a credential helper
  run: |
    gcloud auth configure-docker

You can also launch actions as a module with "uses":

- name: Checkout
  uses: actions/checkout@v2

By default, all jobs run in parallel, and the steps in them are done in sequence. But by using "needs", you can specify that one job should wait for the rest to complete:

needs:
- gcloud-setup-and-build-and-publish-to-GCR
- gke-provisioner

By the way, in the GitHub Web interface, such waiting jobs appear only when the jobs they’re waiting for are executed.

The "gke-provisioner" job mentions Terraform, which we examined in the previous article. The preliminary settings for its operation in the GCP environment are repeated for convenience in a separate markdown file. Here are some additional useful links (UPD - this GitHub Actioh has been superseded by hashicorp/setup-terraform from the moment of initial writing):

In the "kubernetes-deploy" job, there is a step called "Apply Kubernetes manifests". We’re going to use manifests as mentioned in the article Deploying InterSystems IRIS Solution into GCP Kubernetes Cluster GKE Using CircleCI, but with a slight change.

In the previous articles, IRIS application has been stateless. That is, when restarting the pod, all data is returned to its default place. This is great, and it’s often necessary, but for ZPM registry you need to somehow save the packages that were loaded into it, regardless of how many times you need to restart. Deployment allows you to do this, of course, but not without limitations.

For stateful applications, it’s better to choose the StatefulSet resource. Pros and cons can be found in the GKE documentation topic on Deployments vs. StatefulSets and the blog post Kubernetes Persistent Volumes with Deployment and StatefulSet.

The StatefulSet resource is in the repository. Here’s the part that’s important for us:

volumeClaimTemplates:
- metadata:
    name: zpm-registry-volume
    namespace: iris
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi

The code creates a 10GB read/write disk that can be mounted by a single Kubernetes worker node. This disk (and the data on it) will survive the restart of the application. It can also survive the removal of the entire StatefulSet, but for this you need to set the correct Reclaim Policy, which we won’t cover here.

Before breathing life into our workflow, let's add a few more variables to GitHub Secrets:

The following table explains the meaning of these settings (service account keys are also present):

Name	Meaning	Example
GCR_LOCATION	Global GCR location	eu.gcr.io
GKE_CLUSTER	GKE cluster name	dev-cluster
GKE_ZONE	Zone to store an image	europe-west1-b
IMAGE_NAME	Image registry name	zpm-registry
PROJECT_ID	GCP Project ID	possible-symbol-254507
SERVICE_ACCOUNT_KEY	JSON key GitHub uses to connect to GCP. Important: it has to be base64-encoded (see note below)	ewogICJ0eXB...
TF_SERVICE_ACCOUNT_KEY	JSON key Terraform uses to connect to GCP (see note below)	{ ... }

For SERVICE_ACCOUNT_KEY, if your JSON-key has a name, for instance, key.json, run the following command:

$ base64 key.json | tr -d '\n'

For TF_SERVICE_ACCOUNT_KEY, note that its rights are described in Automating GKE creation on CircleCI builds.

One small note about SERVICE_ACCOUNT_KEY: if you, like me, initially forgot to convert it to base64 format, you’ll see a screen like this:

Now that we’ve looked at the workflow backbone and added the necessary variables, we’re ready to examine the full version of the workflow (<root_repo_dir>/.github/workflow/workflow.yaml):

name: Build ZPM-registry image, deploy it to GCR. Run GKE. Run ZPM-registry in GKE
on:
  push:
    branches:
    - master
# Environment variables.
# ${{ secrets }} are taken from GitHub -> Settings -> Secrets
# ${{ github.sha }} is the commit hash
env:
  PROJECT_ID: ${{ secrets.PROJECT_ID }}
  SERVICE_ACCOUNT_KEY: ${{ secrets.SERVICE_ACCOUNT_KEY }}
  GOOGLE_CREDENTIALS: ${{ secrets.TF_SERVICE_ACCOUNT_KEY }}
  GITHUB_SHA: ${{ github.sha }}
  GCR_LOCATION: ${{ secrets.GCR_LOCATION }}
  IMAGE_NAME: ${{ secrets.IMAGE_NAME }}
  GKE_CLUSTER: ${{ secrets.GKE_CLUSTER }}
  GKE_ZONE: ${{ secrets.GKE_ZONE }}
  K8S_NAMESPACE: iris
  STATEFULSET_NAME: zpm-registry

jobs:
  gcloud-setup-and-build-and-publish-to-GCR:
    name: Setup gcloud utility, Build ZPM image and Publish it to Container Registry
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Setup gcloud cli
      uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        version: '275.0.0'
        service_account_key: ${{ secrets.SERVICE_ACCOUNT_KEY }}

    - name: Configure docker to use the gcloud as a credential helper
      run: |
        gcloud auth configure-docker

    - name: Build ZPM image
      run: |
        docker build -t ${GCR_LOCATION}/${PROJECT_ID}/${IMAGE_NAME}:${GITHUB_SHA} .

    - name: Publish ZPM image to Google Container Registry
      run: |
        docker push ${GCR_LOCATION}/${PROJECT_ID}/${IMAGE_NAME}:${GITHUB_SHA}

  gke-provisioner:
  # Inspired by:
  ## https://www.terraform.io/docs/github-actions/getting-started.html
  ## https://github.com/hashicorp/terraform-github-actions
    name: Provision GKE cluster
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Terraform init
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.17
        tf_actions_subcommand: 'init'
        tf_actions_working_dir: 'terraform'

    - name: Terraform validate
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.17
        tf_actions_subcommand: 'validate'
        tf_actions_working_dir: 'terraform'

    - name: Terraform plan
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.17
        tf_actions_subcommand: 'plan'
        tf_actions_working_dir: 'terraform'

    - name: Terraform apply
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.17
        tf_actions_subcommand: 'apply'
        tf_actions_working_dir: 'terraform'

  kubernetes-deploy:
    name: Deploy Kubernetes manifests to GKE cluster
    needs:
    - gcloud-setup-and-build-and-publish-to-GCR
    - gke-provisioner
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Replace placeholders with values in statefulset template
      working-directory: ./k8s/
      run: |
        cat statefulset.tpl |\
        sed "s|DOCKER_REPO_NAME|${GCR_LOCATION}/${PROJECT_ID}/${IMAGE_NAME}|" |\
        sed "s|DOCKER_IMAGE_TAG|${GITHUB_SHA}|" > statefulset.yaml
        cat statefulset.yaml

    - name: Setup gcloud cli
      uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
      with:
        version: '275.0.0'
        service_account_key: ${{ secrets.SERVICE_ACCOUNT_KEY }}

    - name: Apply Kubernetes manifests
      working-directory: ./k8s/
      run: |
        gcloud container clusters get-credentials ${GKE_CLUSTER} --zone ${GKE_ZONE} --project ${PROJECT_ID}
        kubectl apply -f namespace.yaml
        kubectl apply -f service.yaml
        kubectl apply -f statefulset.yaml
        kubectl -n ${K8S_NAMESPACE} rollout status statefulset/${STATEFULSET_NAME}

Before you push to a repository, you should take the terraform-code from the Terraform directory of github-gke-zpm-registry repository, replace placeholders as noted in main.tf comment, and put it inside the terraform/ directory. Remember that Terraform uses a remote bucket that should be initially created as noted in Automating GKE creation on CircleCI builds article.

Also, Kubernetes-code should be taken from the K8S directory of github-gke-zpm-registry repository and put inside the k8s/ directory. These code sources were omitted in this article to save space.

Then you can trigger a deploy:

$ cd <root_repo_dir>/
$ git add .github/workflow/workflow.yaml k8s/ terraform/
$ git commit -m “Add GitHub Actions deploy”
$ git push

After pushing the changes to our forked ZPM repository, we can take a look at the implementation of the steps we described:

There are only two jobs so far. The third, "kubernetes-deploy", will appear after the completion of those on which it depends.
Note that building and publishing Docker images requires some time:

And you can check the result in the GCR console:

The "Provision GKE cluster" job takes longer the first time as it creates the GKE cluster. You’ll see a waiting screen for a few minutes:

But, finally, it finishes and you can be happy:

The Kubernetes resources are also happy:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <GKE_ZONE> --project <PROJECT_ID>
$ kubectl get nodes
NAME                                                                                                   STATUS ROLES   AGE        VERSION
gke-dev-cluster-dev-cluster-node-pool-98cef283-dfq2 Ready    <none> 8m51s   v1.13.11-gke.23

$ kubectl -n iris get po
NAME            READY  STATUS   RESTARTS   AGE
zpm-registry-0  1/1    Running  0          8m25s

It's a good idea to wait for Running status, then check other things:

$ kubectl -n iris get sts
NAME           READY   AGE
zpm-registry   1/1     8m25s


$ kubectl -n iris get svc
NAME           TYPE            CLUSTER-IP       EXTERNAL-IP    PORT(S)                       AGE
zpm-registry   LoadBalancer    10.23.248.234   104.199.6.32     52773:32725/TCP               8m29s

Even the disks are happy:

$ kubectl get pv -oyaml | grep pdName
  pdName: gke-dev-cluster-5fe434-pvc-5db4f5ed-4055-11ea-a6ab-42010af00286

And happiest of all is the ZPM registry (we took the External-IP output of "kubectl -n iris get svc"):

$ curl -u _system:SYS 104.199.6.32:52773/registry/_ping
{"message":"ping"}

Handling the login/password over HTTP is a shame, but I hope to do something about this in future articles.

By the way, you can find more information about endpoints in the source code: see the XData UrlMap section.

We can test this repo by pushing a package to it. There’s a cool ability to push just a direct GitHub link. Let’s try with the math library for InterSystems ObjectScript. Run this from your local machine:

$ curl -XGET -u _system:SYS 104.199.6.32:52773/registry/packages/-/all
[]
$ curl -i -XPOST -u _system:SYS -H "Content-Type: application/json" -d '{"repository":"https://github.com/psteiwer/ObjectScript-Math"}' 'http://104.199.6.32:52773/registry/package'
HTTP/1.1 200 OK
$ curl -XGET -u _system:SYS 104.199.6.32:52773/registry/packages/-/all
[{"name":"objectscript-math","versions":["0.0.4"]}]

Restart a pod to be sure that the data is in place:

$ kubectl -n iris scale --replicas=0 sts zpm-registry
$ kubectl -n iris scale --replicas=1 sts zpm-registry
$ kubectl -n iris get po -w

Wait for a running pod. Then what I hope you’ll see:

$ curl -XGET -u _system:SYS 104.199.6.32:52773/registry/packages/-/all
[{"name":"objectscript-math","versions":["0.0.4"]}]

Let’s install this math package from your repository on your local IRIS instance. Choose the one where the ZPM client is already installed:


$ docker exec -it $(docker run -d intersystemsdc/iris-community:2019.4.0.383.0-zpm) bash

$ iris session iris
USER>write ##class(Math.Math).Factorial(5)
 *Math.Math

USER>zpm
zpm: USER>list

zpm: USER>repo -list
registry
    Source:     https://pm.community.intersystems.com
    Enabled?    Yes
    Available?    Yes
    Use for Snapshots?    Yes
    Use for Prereleases?    Yes

zpm: USER>repo -n registry -r -url http://104.199.6.32:52773/registry/ -user _system -pass SYS

zpm: USER>repo -list                                                                          
registry
    Source:     http://104.199.6.32:52773/registry/
    Enabled?    Yes
    Available?    Yes
    Use for Snapshots?    Yes
    Use for Prereleases?    Yes
    Username:     _system
    Password:     ***

zpm: USER>repo -list-modules -n registry
objectscript-math 0.0.4

zpm: USER>install objectscript-math
[objectscript-math]    Reload START
...
[objectscript-math]    Activate SUCCESS

zpm: USER>quit

USER>write ##class(Math.Math).Factorial(5)                                               
120

Congratulations!
Don’t forget to remove the GKE cluster when you don’t need it anymore:

Conclusion

There are not many references to GitHub Actions within the InterSystems community. I found only one mention from guru @mdaimor. But GitHub Actions can be quite useful for developers storing code on GitHub. Native actions supported only in JavaScript, but this could be dictated by a desire to describe steps in code, which most developers are familiar with. In any case, you can use Docker actions if you don’t know JavaScript.

Regarding the GitHub Actions UI, along the way I discovered a couple of inconveniences that you should be aware of:

You cannot check what is going on until a job step is finished. It’s not clickable, like in the step "Terraform apply".
While you can rerun a failed workflow, I didn’t find a way to rerun a successful workflow.
A workaround for the second point is to use the command:

$ git commit --allow-empty -m "trigger GitHub actions"

You can learn more about this in the StackOverflow question How do I re-run Github Actions?

Continue reading with the next part: Adding TLS and DNS to IRIS-based Services Deployed on Google Kubernetes Engine.

Deploying an InterSystems IRIS Solution on EKS using GitHub Actions

Mikhail Khomenko — Thu, 09 Jul 2020 20:40:43 +0000

Imagine you want to see what InterSystems can give you in terms of data analytics. You studied the theory and now you want some practice. Fortunately, InterSystems provides a project that contains some good examples: Samples BI. Start with the README file, skipping anything associated with Docker, and go straight to the step-by-step installation. Launch a virtual instance, install IRIS there, follow the instructions for installing Samples BI, and then impress the boss with beautiful charts and tables. So far so good.
Inevitably, though, you’ll need to make changes.

It turns out that keeping a virtual machine on your own has some drawbacks, and it’s better to keep it with a cloud provider. Amazon seems solid, and you create an AWS account (free to start), read that using the root user identity for everyday tasks is evil, and create a regular IAM user with admin permissions.

Clicking a little, you create your own VPC network, subnets, and a virtual EC2 instance, and also add a security group to open the IRIS web port (52773) and ssh port (22) for yourself. Repeat the installation of IRIS and Samples BI. This time, use Bash scripting, or Python if you prefer. Again, impress the boss.

But the ubiquitous DevOps movement leads you to start reading about Infrastructure as Code and you want to implement it. You choose Terraform, since it’s well-known to everyone and its approach is quite universal—suitable with minor adjustments for various cloud providers. You describe the infrastructure in HCL language, and translate the installation steps for IRIS and Samples BI to Ansible. Then you create one more IAM user to enable Terraform to work. Run it all. Get a bonus at work.

Gradually you come to the conclusion that in our age of microservices it’s a shame not to use Docker, especially since InterSystems tells you how. You return to the Samples BI installation guide and read the lines about Docker, which don’t seem to be complicated:

$ docker pull intersystemsdc/iris-community:2019.4.0.383.0-zpm
$ docker run --name irisce -d --publish 52773:52773 intersystemsdc/iris-community:2019.4.0.383.0-zpm
$ docker exec -it irisce iris session iris
USER>zpm
zpm: USER>install samples-bi

After directing your browser to http://localhost:52773/csp/user/_DeepSee.UserPortal.Home.zen?$NAMESPACE=USER, you again go to the boss and get a day off for a nice job.

You then begin to understand that “docker run” is just the beginning, and you need to use at least docker-compose. Not a problem:

$ cat docker-compose.yml
version: "3.7"
services:
  irisce:
    container_name: irisce
    image: intersystemsdc/iris-community:2019.4.0.383.0-zpm
    ports:
    - 52773:52773
$ docker rm -f irisce # We don’t need the previous container
$ docker-compose up -d

So you install Docker and docker-compose with Ansible, and then just run the container, which will download an image if it’s not already present on the machine. Then you install Samples BI.

You certainly like Docker, because it’s a cool and simple interface to various kernel stuff. You start using Docker elsewhere and often launch more than one container. And find that often containers must communicate with each other, which leads to reading about how to manage multiple containers.

And you come to Kubernetes.

One option to quickly switch from docker-compose to Kubernetes is to use kompose. Personally, I prefer to simply copy Kubernetes manifests from manuals and then edit for myself, but kompose does a good job of completing its small task:

$ kompose convert -f docker-compose.yml
INFO Kubernetes file "irisce-service.yaml" created
INFO Kubernetes file "irisce-deployment.yaml" created

Now you have the deployment and service files that can be sent to some Kubernetes cluster. You find out that you can install a minikube, which lets you run a single-node Kubernetes cluster and is just what you need at this stage. After a day or two of playing with the minikube sandbox, you’re ready to use a real live Kubernetes deployment somewhere in the AWS cloud.

Getting Set Up

So, let’s do this together. At this point we'll make a couple assumptions:
First, we assume you have an AWS account, you know its ID, and you don’t use root credentials. You create an IAM user (let's call it "my-user") with administrator rights and programmatic access only and store its credentials. You also create another IAM user, called "terraform", with the same permissions:

On its behalf, Terraform will go to your AWS account and create and delete the necessary resources. The extensive rights of both users are explained by the fact that this is a demo. You save credentials locally for both IAM users:

$ cat ~/.aws/credentials
[terraform]
aws_access_key_id = ABCDEFGHIJKLMNOPQRST
aws_secret_access_key = ABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890123
[my-user]
aws_access_key_id = TSRQPONMLKJIHGFEDCBA
aws_secret_access_key = TSRQPONMLKJIHGFEDCBA01234567890123

Note: Don’t copy and paste the credentials from above. They are provided here as an example and no longer exist. Edit the ~/.aws/credentials file and introduce your own records.

Second, we’ll use the dummy AWS Account ID (01234567890) for the article, and the AWS region “eu-west-1.” Feel free to use another region.

Third, we assume you’re aware that AWS is not free and you’ll have to pay for resources used.

Next, you’ve installed the AWS CLI utility for command-line communication with AWS. You can try to use aws2, but you’ll need to specifically set aws2 usage in your kube config file, as described here.

You’ve also installed the kubectl utility for command-line communication with AWS Kubernetes.

And you’ve installed the kompose utility for docker-compose.yml for converting Kubernetes manifests.

Finally, you’ve created an empty GitHub repository and cloned it to your host. We’ll refer to its root directory as <root_repo_dir>. In this repository, we’ll create and fill three directories: .github/workflows/, k8s/, and terraform/.

Note that all the relevant code is duplicated in the github-eks-samples-bi repo to simplify copying and pasting.
Let’s continue.

AWS EKS Provisioning

We already met EKS in the article Deploying a Simple IRIS-Based Web Application Using Amazon EKS. At that time, we created a cluster semi-automatically. That is, we described the cluster in a file, and then manually launched the eksctl utility from a local machine, which created the cluster according to our description.

eksctl was developed for creating EKS clusters and it’s good for a proof-of-concept implementation, but for everyday usage it’s better to use something more universal, such as Terraform. A great resource, AWS EKS Introduction, explains the Terraform configuration needed to create an EKS cluster. An hour or two spent getting acquainted with it will not be a waste of time.

You can play with Terraform locally. To do so, you’ll need a binary (we’ll use the latest version for Linux at the initial time of writing of the article, 0.12.20, Update - the newer version 0.13 was announced recently), and the IAM user "terraform" with sufficient rights for Terraform to go to AWS. Create the directory <root_repo_dir>/terraform/ to store Terraform code:

$ mkdir <root_repo_dir>/terraform
$ cd <root_repo_dir>/terraform

You can create one or more .tf files (they are merged at startup). Just copy and paste the code examples from AWS EKS Introduction and then run something like:

$ export AWS_PROFILE=terraform
$ export AWS_REGION=eu-west-1
$ terraform init
$ terraform plan -out eks.plan

You may encounter some errors. If so, play a little with debug mode, but remember to turn it off later:

$ export TF_LOG=debug
$ terraform plan -out eks.plan
<many-many lines here>
$ unset TF_LOG

This experience will be useful, and most likely you’ll get an EKS cluster launched (use “terraform apply” for that). Check it out in the AWS console:

Clean up when you get bored:

$ terraform destroy

Then go to the next level and start using the Terraform EKS module, especially since it’s based on the same EKS introduction. In the examples/ directory you’ll see how to use it. You’ll also find other examples there.

We simplified the examples somewhat. Here’s the main file in which the VPC creation and EKS creation modules are called:

$ cat <root_repo_dir>/terraform/main.tf
terraform {
  required_version = ">= 0.12.0"
  backend "s3" {
    bucket         = "eks-github-actions-terraform"
    key            = "terraform-dev.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "eks-github-actions-terraform-lock"
  }
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
  version                = "1.10.0"
}

locals {
  vpc_name             = "dev-vpc"
  vpc_cidr             = "10.42.0.0/16"
  private_subnets      = ["10.42.1.0/24", "10.42.2.0/24"]
  public_subnets       = ["10.42.11.0/24", "10.42.12.0/24"]
  cluster_name         = "dev-cluster"
  cluster_version      = "1.14"
  worker_group_name    = "worker-group-1"
  instance_type        = "t2.medium"
  asg_desired_capacity = 1
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

data "aws_availability_zones" "available" {
}

module "vpc" {
  source               = "git::https://github.com/terraform-aws-modules/terraform-aws-vpc?ref=master"

  name                 = local.vpc_name
  cidr                 = local.vpc_cidr
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = local.private_subnets
  public_subnets       = local.public_subnets
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb" = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb" = "1"
  }
}

module "eks" {
  source = "git::https://github.com/terraform-aws-modules/terraform-aws-eks?ref=master"
  cluster_name     = local.cluster_name
  cluster_version  = local.cluster_version
  vpc_id           = module.vpc.vpc_id
  subnets          = module.vpc.private_subnets
  write_kubeconfig = false

  worker_groups = [
    {
      name                 = local.worker_group_name
      instance_type        = local.instance_type
      asg_desired_capacity = local.asg_desired_capacity
    }
  ]

  map_accounts = var.map_accounts
  map_roles    = var.map_roles
  map_users    = var.map_users
}

Let’s look a little more closely at the "terraform" block in main.tf:

terraform {
  required_version = ">= 0.12.0"
  backend "s3" {
    bucket         = "eks-github-actions-terraform"
    key            = "terraform-dev.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "eks-github-actions-terraform-lock"
  }
}

Here we indicate that we’ll adhere to the syntax not lower than Terraform 0.12 (much has changed compared with earlier versions), and also that Terraform shouldn’t store its state locally, but rather remotely, in the S3 bucket.

It’s convenient if the terraform code can be updated from different places by different people, which means we need to be able to lock a user’s state, so we added a lock using a dynamodb table. Read more about locks on the State Locking page.

Since the name of the bucket should be unique throughout AWS, the name “eks-github-actions-terraform” won’t work for you. Please think up your own and make sure it’s not already taken (so you’re getting a NoSuchBucket error):

$ aws s3 ls s3://my-bucket
An error occurred (AllAccessDisabled) when calling the ListObjectsV2 operation: All access to this object has been disabled
$ aws s3 ls s3://my-bucket-with-name-that-impossible-to-remember
An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

Having come up with a name, create the bucket (we use the IAM user "terraform" here. It has administrator rights so it can create a bucket) and enable versioning for it (which will save your nerves in the event of a configuration error):

$ aws s3 mb s3://eks-github-actions-terraform --region eu-west-1
make_bucket: eks-github-actions-terraform
$ aws s3api put-bucket-versioning --bucket eks-github-actions-terraform --versioning-configuration Status=Enabled
$ aws s3api get-bucket-versioning --bucket eks-github-actions-terraform
{
  "Status": "Enabled"
}

With DynamoDB, uniqueness is not needed, but you do need to create a table first:

$ aws dynamodb create-table                       \
  --region eu-west-1                              \
  --table-name eks-github-actions-terraform-lock  \
  --attribute-definitions AttributeName=LockID,AttributeType=S              \
  --key-schema AttributeName=LockID,KeyType=HASH  \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Keep in mind that, in case of Terraform failure, you may need to remove a lock manually from the AWS console. But be careful when doing so.

With regard to the module eks/vpc blocks in main.tf, the way to reference the module available on GitHub is simple:

git::https://github.com/terraform-aws-modules/terraform-aws-vpc?ref=master

Now let’s look at our other two Terraform files (variables.tf and outputs.tf). The first holds our Terraform variables:

$ cat <root_repo_dir>/terraform/variables.tf
variable "region" {
  default = "eu-west-1"
}

variable "map_accounts" {
  description = "Additional AWS account numbers to add to the aws-auth configmap. See examples/basic/variables.tf for example format."
  type        = list(string)
  default     = []
}

variable "map_roles" {
  description = "Additional IAM roles to add to the aws-auth configmap."
  type = list(object({
    rolearn  = string
    username = string
    groups   = list(string)
  }))
  default = []
}

variable "map_users" {
  description = "Additional IAM users to add to the aws-auth configmap."
  type = list(object({
    userarn  = string
    username = string
    groups   = list(string)
  }))
  default = [
    {
      userarn  = "arn:aws:iam::01234567890:user/my-user"
      username = "my-user"
      groups   = ["system:masters"]
    }
  ]
}

The most important part here is adding the IAM user "my-user" to the map_users variable, but you should use your own account ID here in place of 01234567890.

What does this do? When you communicate with EKS through the local kubectl client, it sends requests to the Kubernetes API server, and each request goes through authentication and authorization processes so Kubernetes can understand who sent the request and what they can do. So the EKS version of Kubernetes asks AWS IAM for help with user authentication. If the user who sent the request is listed in AWS IAM (we pointed to his ARN here), the request goes to the authorization stage, which EKS processes itself, but according to our settings. Here, we indicated that the IAM user "my-user" is very cool (group "system: masters").

Finally, the outputs.tf file describes what Terraform should print after it finishes a job:

$ cat <root_repo_dir>/terraform/outputs.tf
output "cluster_endpoint" {
  description = "Endpoint for EKS control plane."
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane."
  value       = module.eks.cluster_security_group_id
}

output "config_map_aws_auth" {
  description = "A kubernetes configuration to authenticate to this EKS cluster."
  value       = module.eks.config_map_aws_auth
}

This completes the description of the Terraform part. We’ll return soon to see how we’re going to launch these files.

Kubernetes Manifests

So far, we’ve taken care of where to launch the application. Now let’s look at what to run.

Recall that we have docker-compose.yml (we renamed the service and added a couple of labels that kompose will use shortly) in the <root_repo_dir>/k8s/ directory:


$ cat /k8s/docker-compose.yml
version: "3.7"
services:
  samples-bi:
    container_name: samples-bi
    image: intersystemsdc/iris-community:2019.4.0.383.0-zpm
    ports:
    - 52773:52773
    labels:
      kompose.service.type: loadbalancer
      kompose.image-pull-policy: IfNotPresent

Run kompose and then add what’s highlighted below. Delete annotations (to make things more intelligible):


$ kompose convert -f docker-compose.yml --replicas=1
$ cat /k8s/samples-bi-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    io.kompose.service: samples-bi
  name: samples-bi
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        io.kompose.service: samples-bi
    spec:
      containers:
      - image: intersystemsdc/iris-community:2019.4.0.383.0-zpm
        imagePullPolicy: IfNotPresent
        name: samples-bi
        ports:
        - containerPort: 52773
        resources: {}
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/bash
              - -c
              - |
                echo -e "write\nhalt" > test
                until iris session iris < test; do sleep 1; done
                echo -e "zpm\ninstall samples-bi\nquit\nhalt" > samples_bi_install
                iris session iris < samples_bi_install
                rm test samples_bi_install
        restartPolicy: Always

We use the Recreate update strategy, which means that the pod will be deleted first and then recreated. This is permissible for demo purposes and allows us to use fewer resources.
We also added the postStart hook, which will trigger immediately after the pod starts. We wait until IRIS starts up and install the samples-bi package from the default zpm-repository.
Now we add the Kubernetes service (also without annotations):

$ cat <root_repo_dir>/k8s/samples-bi-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    io.kompose.service: samples-bi
  name: samples-bi
spec:
  ports:
  - name: "52773"
    port: 52773
    targetPort: 52773
  selector:
    io.kompose.service: samples-bi
  type: LoadBalancer

Yes, we’ll deploy in the "default" namespace, which will work for the demo.

Okay, now we know where and what we want to run. It remains to see how.

The GitHub Actions Workflow

Rather than doing everything from scratch, we’ll create a workflow similar to the one described in Deploying InterSystems IRIS solution on GKE Using GitHub Actions. This time we don’t have to worry about building a container. The GKE-specific parts are replaced by those specific to EKS. Bolded parts are related to receiving the commit message and using it in conditional steps:

$ cat <root_repo_dir>/.github/workflows/workflow.yaml
name: Provision EKS cluster and deploy Samples BI there
on:
  push:
    branches:
    - master

# Environment variables.
# ${{ secrets }} are taken from GitHub -> Settings -> Secrets
# ${{ github.sha }} is the commit hash
env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_REGION: ${{ secrets.AWS_REGION }}
  CLUSTER_NAME: dev-cluster
  DEPLOYMENT_NAME: samples-bi

jobs:
  eks-provisioner:
    # Inspired by:
    ## https://www.terraform.io/docs/github-actions/getting-started.html
    ## https://github.com/hashicorp/terraform-github-actions
    name: Provision EKS cluster
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Get commit message
      run: |
        echo ::set-env name=commit_msg::$(git log --format=%B -n 1 ${{ github.event.after }})

    - name: Show commit message
      run: echo $commit_msg

    - name: Terraform init
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.20
        tf_actions_subcommand: 'init'
        tf_actions_working_dir: 'terraform'

    - name: Terraform validate
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.20
        tf_actions_subcommand: 'validate'
        tf_actions_working_dir: 'terraform'

    - name: Terraform plan
      if: "!contains(env.commit_msg, '[destroy eks]')"
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.20
        tf_actions_subcommand: 'plan'
        tf_actions_working_dir: 'terraform'

    - name: Terraform plan for destroy
      if: "contains(env.commit_msg, '[destroy eks]')"
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.20
        tf_actions_subcommand: 'plan'
        args: '-destroy -out=./destroy-plan'
        tf_actions_working_dir: 'terraform'

    - name: Terraform apply
      if: "!contains(env.commit_msg, '[destroy eks]')"
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.20
        tf_actions_subcommand: 'apply'
        tf_actions_working_dir: 'terraform'

    - name: Terraform apply for destroy
      if: "contains(env.commit_msg, '[destroy eks]')"
      uses: hashicorp/terraform-github-actions@master
      with:
        tf_actions_version: 0.12.20
        tf_actions_subcommand: 'apply'
        args: './destroy-plan'
        tf_actions_working_dir: 'terraform'

  kubernetes-deploy:
    name: Deploy Kubernetes manifests to EKS
    needs:
    - eks-provisioner
    runs-on: ubuntu-18.04
    steps:
    - name: Checkout
      uses: actions/checkout@v2

    - name: Get commit message
      run: |
        echo ::set-env name=commit_msg::$(git log --format=%B -n 1 ${{ github.event.after }})

    - name: Show commit message
      run: echo $commit_msg

    - name: Configure AWS Credentials
      if: "!contains(env.commit_msg, '[destroy eks]')"
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ${{ secrets.AWS_REGION }}

    - name: Apply Kubernetes manifests
      if: "!contains(env.commit_msg, '[destroy eks]')"
      working-directory: ./k8s/
      run: |
        aws eks update-kubeconfig --name ${CLUSTER_NAME}
        kubectl apply -f samples-bi-service.yaml
        kubectl apply -f samples-bi-deployment.yaml
        kubectl rollout status deployment/${DEPLOYMENT_NAME}

Of course, we need to set the credentials of the "terraform" user (take them from the ~/.aws/credentials file), letting Github use its secrets:

Notice that workflow will enable us to destroy an EKS cluster by pushing a commit message that contains a phrase "[destroy eks]". Also note that we won’t run "kubernetes apply" with such a commit message.
Run a pipeline, but first create a .gitignore file:

$ cat <root_repo_dir>/.gitignore
.DS_Store
terraform/.terraform/
terraform/*.plan
terraform/*.json
$ cd <root_repo_dir>
$ git add .github/ k8s/ terraform/ .gitignore
$ git commit -m "GitHub on EKS"
$ git push

Monitor deployment process on the "Actions" tab of GitHub repository page. Please wait for successful completion.

When you run a workflow for the very first time, it will take about 15 minutes on the "Terraform apply" step, approximately as long as it takes to create the cluster. At the next start (if you didn’t delete the cluster), the workflow will be much faster. You can check this out:

$ cd <root_repo_dir>
$ git commit -m "Trigger" --allow-empty
$ git push

Of course, it would be nice to check what we did. This time you can use the credentials of IAM "my-user" on your laptop:


$ export AWS_PROFILE=my-user
$ export AWS_REGION=eu-west-1
$ aws sts get-caller-identity
$ aws eks update-kubeconfig --region=eu-west-1 --name=dev-cluster --alias=dev-cluster
$ kubectl config current-context
dev-cluster

$ kubectl get nodes
NAME                                                                               STATUS   ROLES      AGE          VERSION
ip-10-42-1-125.eu-west-1.compute.internal   Ready          6m20s     v1.14.8-eks-b8860f

$ kubectl get po
NAME                                                       READY        STATUS      RESTARTS   AGE
samples-bi-756dddffdb-zd9nw    1/1               Running    0                      6m16s

$ kubectl get svc
NAME                   TYPE                        CLUSTER-IP        EXTERNAL-IP                                                                                                                                                         PORT(S)                    AGE
kubernetes        ClusterIP               172.20.0.1                                                                                                                                                                                443/TCP                    11m
samples-bi         LoadBalancer     172.20.33.235    a2c6f6733557511eab3c302618b2fae2-622862917.eu-west-1.elb.amazonaws.com    52773:31047/TCP  6m33s

Go to http://a2c6f6733557511eab3c302618b2fae2-622862917.eu-west-1.elb.amazonaws.com:52773/csp/user/_DeepSee.UserPortal.Home.zen?$NAMESPACE=USER (substitute link by your External-IP), then type "_system", "SYS" and change the default password. You should see a bunch of BI dashboards:

Click on each one’s arrow to deep dive:

Remember, if you restart a samples-bi pod, all your changes will be lost. This is intentional behavior as this is a demo. If you need persistence, I've created an example in the github-gke-zpm-registry/k8s/statefulset.tpl repository.

When you’re finished, just remove everything you’ve created:

$ git commit -m "Mr Proper [destroy eks]" --allow-empty
$ git push

Conclusion

In this article, we replaced the eksctl utility with Terraform to create an EKS cluster. It’s a step forward to “codify” all of your AWS infrastructure.
We showed how you can easily deploy a demo application with git push using Github Actions and Terraform.
We also added kompose and a pod’s postStart hooks to our toolbox.
We didn’t show TLS enabling this time. That’s a task we’ll undertake in the near future.