DEV Community: Bryan Bende

Apache NiFi - Stateless

Bryan Bende — Wed, 10 Nov 2021 00:00:00 +0000

The past several releases of Apache NiFi have made significant improvements to the Stateless NiFi engine. If you are not familiar with Stateless NiFi, then I would recommend reading this overview first.

This post will examine the differences between running a flow in traditional NiFi vs. Stateless NiFi.

Traditional NiFi

As an example, let’s assume there is a Kafka topic with CDC events and we want to consume the events and apply them to a another relational database. This can be achieved with a simple flow containing ConsumeKafka_2_6 connected to PutDatabaseRecord.

In traditional NiFi, each node has a set of internal repositories that are stored on local disk. The _Flow File Repository_contains the state of each flow file, including it’s attributes and location in the flow, and the _Content Repository_stores the content of each flow file.

Each execution of a processor is given a reference to a session that acts like a transaction for operating on flow files. If all operations complete successfully and the session is committed, then all updates are persisted to NiFi’s repositories. In the event that NiFi is restarted, all data is preserved in the repositories and the flow will start processing from the last committed state.

Let’s consider how the example flow will execute in traditional NiFi…

First, ConsumeKafka_2_6 will poll Kafka for available records. Then it will use the session to create a flow file and write the content of the records to the output stream of the flow file. The processor will then commit the NiFi session, followed by committing the Kafka offsets. The flow file will then be transferred to PutDatabaseRecord. The overall sequence is summarized in the following diagram.

A key point here is the ordering of committing the NiFi session before committing the Kafka offsets. This provides an_“at least once”_ guarantee by ensuring the data is persisted in NiFi before acknowledging the offsets. If committing the offsets fails, possibly due to a consumer rebalance, NiFi will consume those same offsets again and receive duplicate data. If the ordering was reversed, it would be possible for the offsets to be successfully committed, followed by a failure to commit the NiFi session, which would create data loss and be considered “at most once”.

A second key point is that there is purposely no coordination across processors, meaning that each processor succeeds or fails independently of the other processors in the flow. Once ConsumeKafka_2_6 successfully executes, the consumed data is now persisted in NiFi, regardless of whether PutDatabaseRecord succeeds.

Let’s look at how this same flow would execute in Stateless NiFi.

Stateless NiFi

Stateless NiFi adheres to the same NiFi API as traditional NiFi, which means it can run the same processors and flow definitions, it just provides a different implementation of the underlying engine.

The primary abstraction is a StatelessDataFlow which can be triggered to execute. Each execution of the flow produces a result that can be considered a success or failure. A failure can occur from a processor throwing an exception, or from explicitly routing flow files to a named “failure port”.

A key difference in Stateless NiFi is around committing the NiFi session. A new commit method was introduced to ProcessSession with the following signature:

void commitAsync(Runnable onSuccess);

This gives the session implementation control over when to execute the given callback. In traditional NiFi the session can execute the callback as the last step of commitAsync, which produces the same behavior we looked at earlier. The stateless NiFi session can hold the callback and execute it only when the entire flow has completed successfully.

Let’s consider how the example flow will execute in Stateless NiFi…

When the StatelessDataFlow is triggered, ConsumeKafka_2_6 begins executing the same as it would in traditional NiFi, by polling Kafka for records, creating a flow file, and writing the records to the output stream of the flow file. It then calls commitAsync to commit the NiFi session and passes in a callback for committing the offsets to Kafka, which in this case will be held until later.

The flow file is then transferred to PutDatabaseRecord which attempts to apply the event to the database. Let’s assumePutDatabaseRecord was successful, then the overall execution of the flow completes successfully. The stateless engine then acknowledges the result which executes any held callbacks, and thus commits the offsets to Kafka. The overall sequence is summarized in the following diagram.

A key point here is that the execution of the entire flow is being treated like a single transaction. If a failure were to occur atPutDatabaseRecord, the overall execution would be considered a failure, and the onSuccess callbacks from commitAsyncwould never get executed. In this case, that would mean the offsets were never committed to Kafka, and the entire flow can be tried again for the same records.

Another type of failure scenario would be if Stateless NiFi crashed in the middle of executing the flow. Since Stateless NiFi generally uses in-memory repositories, any data that was in the middle of processing would be gone. However, since the source processor had not yet acknowledged receiving that data (i.e. the onSuccess callback never got executed), it would pull the same data again on the next execution.

ExecuteStateless

Previously, the primary mechanism to use Stateless NiFi was through the nifi-stateless binary which launches a standalone process to execute the flow.

The 1.15.0 release introduced a new processor called ExecuteStateless which can be used to run the Stateless engine from within traditional NiFi. This allow you to manage the execution of the Stateless flow using the traditional NiFi UI, as well as connect the output of the Stateless flow to follow on processing in the traditional NiFi flow.

In order to use the ExecuteStateless processor, you would first use traditional NiFi to create a process group containing the flow you want to execute with the Stateless engine. You would then download the flow definition, or commit the flow to a NiFi Registry instance. From there, you would configure ExecuteStateless with the location of the flow definition.

For a more in depth look at ExecuteStateless, check out Mark Payne’s YouTube Video on “Kafka Exactly Once with NiFi”.

Apache NiFi 1.14.0 - Secure by Default

Bryan Bende — Mon, 19 Jul 2021 00:00:00 +0000

One of the major improvements in Apache NiFi 1.14.0 was to enable security for the default configuration. This means all you have to do now is run bin/nifi.sh start, and your local instance will be running over https with the ability to login via username and password.

The overall work for this improvement was done through NIFI-8220 and required three major pieces:

Automatic generation of a self-signed certificate
Single User Login Identity Provider
Single User Authorizer

From a high level, the overall setup looks like the following:

Automatic Certificate Generation

In order to have any form of authentication & authorization, we first need to be connecting over https, which means NiFi’s web server needs a keystore and truststore.

In order to achieve this, NIFI-8403 introduced the ability to generate a self-signed certficate during start-up. When keystore and truststore files are specified in nifi.properties and the files don’t exist, they will automatically be generated and nifi.properties will be updated with the passwords.

As a result, the default nifi.propeties file now comes with provided values for the keystore and truststore:

nifi.security.keystore=./conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=./conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=

In addition, the default web host and port have been switched to the following https values:

nifi.web.https.host=127.0.0.1
nifi.web.https.port=8443

As a side note, there are two other new properties related to certificates:

nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs

These were not required for the default secure setup, but they allow the keystore and truststore to be reloaded while the application is running. This can be helpful for replacing certificates that may be close to expiring.

Single User Login Identity Provider

The next step was to provide a mechanism for authenticating the user. NiFi supports many different authentication mechanisms, but most of them require additional dependencies and/or configuration.

In this case, we want a user to login with a username and password without doing anything else. In order to achieve this, NIFI-8363 introduced the Single User Login Identity Provider.

This login identity provider allows a single username/password pair to be configured. When this provider is initialized, if the username and password are not present, random values will be generated and login-identity-providers.xml will be updated with the values.

The default login-identity-providers.xml now contains the following configuration:

<provider>
   <identifier>single-user-provider</identifier>
   <class>org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider</class>
   <property name="Username"></property>
   <property name="Password"></property>
</provider>

NOTE: The password value here is the hashed password. See the last section below about obtaining and changing the default values.

The default nifi.properies then specifies this login identity provider:

nifi.security.user.login.identity.provider=single-user-provider

Single User Authorizer

The next step was providing a mechanism to perform authorization. In this case, we just want the default user to be authorized for all actions. In order to achieve this, NIFI-8363 introduced the Single User Authorizer.

This authorizer just returns true for all authorization checks, with the caveat that it can only be used when the Single User Login Identity Provider is also configured.

The default authorizers.xml now contains the following configuration:

<authorizer>
   <identifier>single-user-authorizer</identifier>
   <class>org.apache.nifi.authorization.single.user.SingleUserAuthorizer</class>
</authorizer>

The default nifi.properties then specifies this authorizer:

nifi.security.user.authorizer=single-user-authorizer

Default Username/Password

The first time the application is started, the Single User Login Identity Provider generates the username and password and logs them to nifi-app.log. An example would be the following:

2021-07-16 15:46:31,006 INFO [main] o.a.n.w.c.ApplicationStartupContextListener Flow Controller started successfully.
2021-07-16 15:46:31,026 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider

Generated Username [6fcaba96-5445-4835-822f-e004c4642d3b]
Generated Password [ScCULiVSEwlqVLG6aHxGv/utRTHxWa7n]

2021-07-16 15:46:31,026 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider Run the following command to change credentials: nifi.sh set-single-user-credentials USERNAME PASSWORD
2021-07-16 15:46:31,338 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider Updating Login Identity Providers Configuration [./conf/login-identity-providers.xml]

If you then access https://localhost:8443/nifi in your browser (accept warnings about self-signed certificates), you should be able to login with the username/password.

As the logs mention above, the default username/password can be changed by running the following utility:

./bin/nifi.sh set-single-user-credentials USERNAME PASSWORD

K3s on Raspberry Pi - Jenkins / Registry (Part 2)

Bryan Bende — Sat, 03 Jul 2021 00:00:00 +0000

This is the second part of setting up Jenkins and a private Docker registry on K3s. The first part left off with the private registry up and running and accessible to K3s, and Jenkins being able to execute a basic job through the Kubernetes plugin.

This post will cover setting up a more realistic Jenkins job for an example Spring Boot application, including publishing images to the private registry and running them on K3s.

Example Application Overview

The code for the example application can be found at cloud-native-examples/example-app.

The project is made up of three modules:

example-app-api - Shared model classes

example-app-backend - Spring Boot application with a single REST endpoint for storing/retrieving messages, messages are stored using a simple in-memory implementation

example-app-frontend - Spring Boot application that communicates with the backend and provides a simple UI to create/view messages

The frontend application.properties contains the location of the backend application:

example.app.backend.server.protocol=http
example.app.backend.server.host=localhost
example.app.backend.server.port=8081
example.app.backend.server.messages-context=/messages

These values would potentially need to be overridden depending how the application is being deployed.

If we start each Spring Boot application locally from Intellij, we should see the following…

Backend

Frontend

Browser at http://localhost:8080

Building Images

Since our goal is to deploy the application to K3s, we need a way to produce container images for the backend and frontend Spring Boot applications.

The spring-boot-maven-plugin recently added support for building OCI images through the new build-image goal. This relies on buildpacks behind the scenes, which unfortunately does not support arm, so the build-image goal won’t succeed from one of the Raspberry Pis.

Another popular option is Jib, a library from Google for containerizing Java applications. Jib has plugins for Maven and Gradle, and has the ability to build images without a local Docker daemon. This is particularly useful for our Jenkins job which is going to execute in a pod that won’t have access to a Docker daemon.

As a side note, if you do need to access a Docker daemon from within a container, a common approach seems to be generally referred to as “Docker in Docker (DIND)”. This boils down to mounting /var/run/docker.sock from the host into the container, so the container is actually using the host’s Docker daemon. Since K3s uses containerd as the runtime, there isn’t a Docker daemon on the host anyway.

The configuration of the Jib plugin is done in pluginManagement at the example-app level so that it can be shared across the frontend and backend applications.

There are three profiles available:

jib-docker-daemon - Builds to the local Docker daemon

jib-docker-hub - Builds and pushes to Docker Hub, requires running docker login first

jib-k3s-private - Builds and pushes to the private registry in K3s, requires specifying username/password of the registry with -Djib.registry.username and -Djib.registry.password

We can test building the images with our local Docker daemon by executing:

mvn clean package -Pjib-docker-daemon

After a successful build, running the command docker images should show the following:

REPOSITORY TAG IMAGE ID CREATED SIZE
example-backend 0.0.1-SNAPSHOT ea3241a5cfd6 25 seconds ago 261MB
example-backend latest ea3241a5cfd6 25 seconds ago 261MB
example-frontend 0.0.1-SNAPSHOT 1cf33e4a9207 37 seconds ago 265MB
example-frontend latest 1cf33e4a9207 37 seconds ago 265MB

Next we can setup a Jenkins job that will use the jib-k3s-private profile to build and publish the images to private registry.

Jenkins Job

First, we need to create a Jenkins Credential to store the username/password for the private registry so that these values can be referenced securely from the build configuration.

Navigate to Manage Jenkins -> Manage Credentials -> “Jenkins” Store -> Domain “Global Credentials (unrestricted)”, and then click Add Credential.

Enter the username and password for the registry, and enter a unique id for the credential, we will call it docker-registry-private. The id can be anything, we just need to reference it later.

Now we can create the new pipeline. From the main Jenkins page, click New Item and add a pipeline named cloud-native-examples. Under the Pipeline Definition, select Pipeline Script and enter the following script:

pipeline {
    environment {
        GIT_REPO_URL = 'https://github.com/bbende/cloud-native-examples.git'
        GIT_REPO_BRANCH = 'main'
        REGISTRY_URL = 'docker-registry-service.docker-registry.svc.cluster.local:5000'
        REGISTRY_CREDENTIAL = credentials('docker-registry-private')
    }
    agent {
        kubernetes {
            defaultContainer 'jnlp'
            yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: maven
      image: maven:3.8.1-openjdk-11
      command: ["tail", "-f", "/dev/null"]
      imagePullPolicy: IfNotPresent
      resources:
        requests:
          memory: "1Gi"
          cpu: "500m"
        limits:
          memory: "1Gi"
      volumeMounts:
            - name: jenkins-maven
              mountPath: /root/.m2
  volumes:
    - name: jenkins-maven
      persistentVolumeClaim:
        claimName: jenkins-maven-pvc
"""
        }
    }
    stages {
        stage('Git Clone') {
            steps {
                git(url: "${GIT_REPO_URL}", branch: "${GIT_REPO_BRANCH}")
            }
        }
        stage('Build') {
            steps {
                container('maven') {
                    sh 'mvn package -Pjib-k3s-private -Djib.registry.username=$REGISTRY_CREDENTIAL_USR -Djib.registry.password=$REGISTRY_CREDENTIAL_PSW -DsendCredentialsOverHttp=true'
                }
            }
        }
    }
}

Let’s breakdown each part of the pipeline…

environment - Defines variables for use in other parts of the pipeline, including a variable for the private registry credential from calling credentials('docker-registry-private')

agent/kubernetes - Defines an agent for executing the pipeline in Kubernetes with a field for the YAML definition of the agent Pod which uses the image maven:3.8.1-openjdk-11

stage(‘Git Clone’) - Clones the git repository for the project

stage(‘Build’) - Executes the Maven build using the jib-k3s-private profile and overrides variables to specify the username/password for the registry using the credential environment variables

As an optimization, the Maven container mounts a persistent volume to /root/.m2 which comes from a Longhorn PVC. This allows the local Maven repository to be persisted across builds, instead of downloading every dependency on every build.

The REGISTRY_URL is defined as docker-registry-service.docker-registry.svc.cluster.local. This is because we have a service named docker-registry-service in the docker-registry namespace, but we are accessing it from Jenkins in the jenkins namespace, so we need the fully qualified hostname.

The registry is secured with username/password authentication, but the internal service at docker-registry-service is not TLS-enabled, so we have to add -DsendCredentialsOverHttp=true to allow Jib to authenticate over http. This is acceptable for internal communication on our example cluster, but is not recommended for a real environment.

After creating the pipeline we can click Build Now to start a build. From watching the Console Output of the build, we should see something like the following:

Created Pod: k3s jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Scheduled] Successfully assigned jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3 to rpi-3
Still waiting to schedule task
‘cloud-native-examples-26-k493m-qzrwp-3xpv3’ is offline
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][SuccessfulAttachVolume] AttachVolume.Attach succeeded for volume "pvc-af0ab285-fe51-48a8-be4c-f106bea22566"
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Pulled] Container image "maven:3.8.1-openjdk-11" already present on machine
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Created] Created container maven
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Started] Started container maven
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Pulled] Container image "pi4k8s/inbound-agent:4.3" already present on machine
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Created] Created container jnlp
[Normal][jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3][Started] Started container jnlp

This shows that the pod jenkins/cloud-native-examples-26-k493m-qzrwp-3xpv3 was launched to execute the build. The persistent volume for the Maven repo was then attached and the containers were created and started.

The set of containers is a combination of the default Pod template created in Part 1, and the YAML definition in the pipeline. So the combined set of containers includes maven:3.8.1-openjdk-11, pi4k8s/inbound-agent:4.3, and jnlp.

The rest of the build should continue with a standard Maven build of a Spring Boot application. At some point in the build output, we should see the frontend and backend images being published:

[INFO Built and pushed image as docker-registry-service.docker-registry.svc.cluster.local:5000/example-frontend, docker-registry-service.docker-registry.svc.cluster.local:5000/example-frontend:0.0.1-SNAPSHOT-k3s
...
[INFO Built and pushed image as docker-registry-service.docker-registry.svc.cluster.local:5000/example-backend, docker-registry-service.docker-registry.svc.cluster.local:5000/example-backend:0.0.1-SNAPSHOT-k3s

We can verify that the images are now available in the private registry by running the following command from our laptop:

curl -k -X GET --basic -u registry https://docker.registry.private/v2/_catalog | python -m json.tool

This shows the following output:

{
    "repositories": [
        "arm32v7/nginx",
        "example-backend",
        "example-frontend"
    ]
}

Testing Images on K3s

Now that the images are available in the private registry, we can test deploying pods on K3s to verify the images work correctly:

kubectl create namespace example-app

kubectl run example-backend --image docker.registry.private/example-backend --namespace example-app

kubectl run example-frontend --image docker.registry.private/example-frontend --namespace example-app

The images are prefixed with the registry location of docker.registry.private, which must line up with the configuration in /etc/rancher/k3s/registries.yaml.

If we check the status of the pods in the example-app namespace, we should see two pods come up running:

kubectl get pods --namespace example-app
NAME READY STATUS RESTARTS AGE
example-backend 1/1 Running 0 102s
example-frontend 1/1 Running 0 59s

This proves K3s is able to pull images from the private registry and the images were correctly built for arm64.

There would be a bit more work to correctly deploy the application. Currently the frontend isn’t correctly configured to access the backend, and the frontend isn’t accessible from outside the cluster, but those are topics for another post.

K3s on Raspberry Pi - Jenkins / Registry (Part 1)

Bryan Bende — Fri, 02 Jul 2021 00:00:00 +0000

Building on my previous K3s posts, this one will cover deploying Jenkins and a private Docker registry, as well as configuring the Jenkins Kubernetes plugin.

The overall setup looks like the following:

Private Registry Setup

For the private registry, I primarily followed this article: Installing Docker Registry on K3s.

My version of the configuration can be found here: ks-config/docker-registry.

In order for K3s to pull images from the private registry, the containerd daemon on each node needs to access the registry running within a pod in K3s.

In my configuration, the registry is exposed via ingress with a host of docker.registry.private, so we can edit /etc/hosts on each node to map this hostname to the current node.

An example from the master node would be the following:

192.168.1.244 rpi-1 docker.registry.private
192.168.1.245 rpi-2
192.168.1.246 rpi-3

This mapping needs to be done on each node since a pod may be deployed to any node in the cluster.

Next we need to configure K3s to know about the private registry. This is done by creating the file /etc/rancher/k3s/registries.yaml on each node:

mirrors:
  docker.registry.private:
    endpoint:
      - "https://docker.registry.private"
configs:
  "docker.registry.private":
    auth:
      username: registry
      password: <replace-with-your-password>
    tls:
      insecure_skip_verify: true

The initial K3s Ansible setup placed everything on the worker nodes under /etc/rancher/node, but registries.yaml is only recognized from /etc/rancher/k3s, so you will need to create this directory first.

I wasn’t sure of the best way to configure the tls section to trust the self-signed certificate presented by registry’s ingress, so specifying insecure_skip_verify: true disables certificate verification for now.

After getting this file in place on each node, you can adjust the permissions and restart K3s.

# Master node
sudo chmod go-r /etc/rancher/k3s/registries.yaml
sudo systemctl restart k3s

# Worker nodes
sudo chmod go-r /etc/rancher/k3s/registries.yaml
sudo systemctl restart k3s-node

At this point, we should be able to deploy a pod that references an image from docker.registry.private. Assuming we pushed the image arm32v7/nginx to our registry, we can deploy the following pod:

apiVersion: v1
kind: Pod
metadata:
  name: docker-registry-test
spec:
  containers:
  - name: nginx
    image: docker.registry.private/arm32v7/nginx

If everything is working correctly, the pod should deploy and end up in the running state.

Jenkins Setup

For deploying Jenkins, I primarily followed these two articles:

My configuration can be found here: ks-config/jenkins

Unfortunately the standard Jenkins image doesn’t support arm64, but the images under jenkins4eval do, so using jenkins4eval/jenkins:latest ended up working.

In my configuration, I created an ingress with the host jenkins.private and then mapped this hostname to the master node in /etc/hosts on my laptop. This follows the same approach for how I’m accessing other services from my laptop, such as Longhorn UI and the private registry.

The first time accessing https://jenkins.private, you will be prompted to unlock Jenkins. The password is printed to the log of the Jenkins pod during start up. The above articles cover this in more detail.

Assuming you are successfully logged in to Jenkins, the next thing to do is install the Kubernetes plugin.

Jenkins Kubernetes Plugin

The Jenkins Kubernetes Plugin launches worker agents as Kubernetes pods, which means we get a fresh build environment for each job based on a pod specification.

From the left hand menu, select “Manage Jenkins” and then “Manage Plugins”. Search for the Kubernetes Plugin and select “Install without Restart”. I already have it installed, but the plugins page looks like this:

After the install completes, restart Jenkins by using the URL https://jenkins.private/safeRestart.

Once Jenkins has restarted, navigate to “Manage Jenkins” -> “Manage Nodes and Clouds” -> “Configure Clouds”, and add a cloud of type Kubernetes.

We need to enter the namespace where Jenkins is deployed, and then we can click Test Connection to verify that Jenkins can communicated with Kubernetes.

In the next section, we need to enter a value for the Jenkins URL field. This is the URL used by the Jenkins worker pod to communicate back to the main Jenkins server. This URL should use the ClusterIP service that we created as part of deploying Jenkins, so the URL should be http://jenkins:8080.

Under the Advanced section, set the Defaults Provider Template Name to default-agent. This is the name of a pod template we are going to create to provide default values for all worker pods. The main reason we are defining a pod template is to override the default jnlp container to a specific one that supports arm64.

Under Pod Templates, click Add Pod Template, enter the name as default-agent and the namespace as jenkins:

Under the Containers section, click Add Container, enter a name of jnlp and specify the image as pi4k8s/inbound-agent:4.3:

We should now have the minimum working configuration for executing a pipeline.

Test Pipeline

From the main Jenkins page, click New Item and add a pipeline named k3s-test-pipleine. Under the Pipeline Definition, select Pipeline Script and enter the following script:

pipeline {
    agent {
        kubernetes {
            defaultContainer 'jnlp'
        }
    }
    stages {
        stage('Hello') {
            steps {
                echo 'Hello everyone! This is now running on a Kubernetes executor!'
            }
        }
    }
}

Click Build Now and we should get a successful execution. The output of the job should print the YAML definition used for the worker pod, which should show the customizations we made:

In the next post I’ll cover how to setup a pipeline that builds an image for a Spring Boot application and publishes to our private registry.

K3s on Raspberry Pi - cert-manager

Bryan Bende — Thu, 01 Jul 2021 00:00:00 +0000

In this post we’ll look at deploying cert-manager on K3s and how to use it with the Traefik Ingress Controller to enable TLS.

Background

In a previous post, we covered the Traefik Ingress Controller and created an example deployment using the whoami container, along with an ingress over http. We can use the IP address of any node, such as http://192.168.1.244/foo, and get a response like the following:

If we change the URL to https, we get a warning about about a self-signed certificate. Clicking View Certificate, shows a self-signed Traefik certificate, and if we accept and continue, we get a 404 Not Found.

This is because the whoami ingress doesn’t have TLS enabled and the request is falling back to Traefik’s default https handling. We can fix this by deploying cert-manager and using it to request a certificate for the whoami ingress.

Deploying cert-manager

The cert-manager documentation covers the options for installing cert-manager on Kubernetes. We are going to use the manifest approach with a slight modification to ensure that arm images are used.

curl -sL \
https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.yaml |\
sed -r 's/(image:.*):(v.*)$/\1-arm:\2/g' > cert-manager-arm.yaml

kubectl create namespace cert-manager
kubectl create -f cert-manager-arm.yaml

This approach came from the article Make SSL certs easy with k3s.

After all of the resources are created, we should eventually see three running pods:

kubectl get pods --namespace cert-manager

NAME READY STATUS
cert-manager-webhook-7c58d9689f-74j7c 1/1 Running
cert-manager-7c5b8cb7cf-nzz64 1/1 Running
cert-manager-cainjector-67df6b6b68-lzkng 1/1 Running

At this point, we can follow the steps from the cert-manager documentation to verify the installation is working correctly.

Traefik Ingress with cert-manager

In order to obtain certificates from cert-manager, we need to create an issuer to act as a certificate authority. We have the option of creating an Issuer which is a namespaced resource, or a ClusterIssuer which is a global resource. We’ll create a self-signed ClusterIssuer using the following definition:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: self-signed-issuer
spec:
  selfSigned: {}

We can then update the whoami ingress to obtain a certificate from the ClusterIssuer and enable TLS. This is done by adding the following annotations:

cert-manager.io/cluster-issuer: self-signed-issuer
traefik.ingress.kubernetes.io/router.tls: "true"

The spec also needs a new tls section to specify the hostname and the name of the secret that will store the certificate created by cert-manager:

tls:
- hosts:
  - whoami
  secretName: whoami-tls

The full ingress definition looks like the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami-tls
  namespace: whoami
  annotations:
    kubernetes.io/ingress.class: "traefik"
    cert-manager.io/cluster-issuer: self-signed-issuer
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  tls:
  - hosts:
    - whoami
    secretName: whoami-tls
  rules:
    - host: whoami
      http:
        paths:
          - path: /bar
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80
          - path: /foo
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80

After creating this ingress, we can inspect the whoami-tls secret:

kubectl --namespace whoami describe secret whoami-tls

Name: whoami-tls
Namespace: whoami
Labels: <none>
Annotations: cert-manager.io/alt-names: whoami
              cert-manager.io/certificate-name: whoami-tls
              cert-manager.io/common-name:
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: self-signed-issuer
              cert-manager.io/uri-sans:

Type: kubernetes.io/tls

Data
====
ca.crt: 1017 bytes
tls.crt: 1017 bytes
tls.key: 1679 bytes

The secret type is kubernetes.io/tls and the data holds the CA certificate, the whoami public certificate (tls.crt), and the whoami private key (tls.key).

In order to access the whoami service via https, the hostname of the URL must match the hostname of the certificate, which in this case is whoami. To make this work, we can modify /etc/hosts so that whoami maps to the IP address of one of our nodes.

192.168.1.244 rpi-1 whoami
192.168.1.245 rpi-2
192.168.1.246 rpi-3

Now if we access https://whoami/bar in our browser, we still get a warning about a self-signed certificate, but this time the certificate is the whoami certificate and not the default Traefik certificate.

If we accept the warning and continue, we now get a successful response!

K3s on Raspberry Pi - Volumes and Storage

Bryan Bende — Sat, 15 May 2021 00:00:00 +0000

In this post we’ll look at how volumes and storage work in a K3s cluster.

Local Path Provisioner

K3s comes with a default Local Path Provisioner that allows creating a PersistentVolumeClaim backed by host-based storage. This means the volume is using storage on the host where the pod is located.

Let’s take a look at an example…

Create a specification for a PersistentVolumeClaim and use the storageClassName of local-path:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: local-path-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi

Create a specification for a Pod that binds to this PVC:

apiVersion: v1
kind: Pod
metadata:
  name: local-path-test
  namespace: default
spec:
  containers:
  - name: local-path-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: local-path-pvc
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: local-path-pvc
    persistentVolumeClaim:
      claimName: local-path-pvc

Create these two resources with kubectl:

kubectl create -f pvc-local-path.yaml
kubectl create -f pod-local-path-test.yaml

After a minute or so, we should be able to see our running pod:

kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE   
local-path-test 1/1 Running 0 7m7s 10.42.2.34 rpi-2

The pod only has one container, so we can get a shell to the container with the following command:

kubectl exec -it local-path-test -- sh
/ #

Create a test file in /data, which is the mountPath for the persistent volume:

/ # echo "testing" > /data/test.txt
/ # ls -l /data/
total 4
-rw-r--r-- 1 root root 8 May 15 20:24 test.txt

The output above showed that this pod is running on the node rpi-2.

If we exit out of the shell for the container, and then SSH to the node rpi-2, we can expect to the find the file test.txt somewhere, since the persistent volume is backed by host-based storage.

ssh pi@rpi-2
sudo find / -name test.txt
/var/lib/rancher/k3s/storage/pvc-a5a439ab-ea94-481d-bee2-9e5c0e0f8008_default_local-path-pvc/test.txt

This shows that we get out-of-the-box support for persistent storage with K3s, but what if our pod goes down and is relaunched on a different node?

In that case, the new node won’t have access to the data from /var/lib/rancher/k3s/storage on the previous node. To handle that case, we need distributed block storage.

With distributed block storage, the storage is decouple from the pods, and the PersistentVolumeClaim can be mounted to the pod regardless of where the pod is running.

Longhorn

Longhorn is a “lightweight, reliable and easy-to-use distributed block storage system for Kubernetes.” It is also created by Rancher Labs, so it makes the integration with K3s very easy.

For the most part, we can just follow the Longhorn Install Documentation.

First, make sure you install the open-iscsi package on all nodes:

sudo apt-get install open-iscsi

On my first attempt, I forgot to install this package and ran into issues later.

Deploy Longhorn using kubectl:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v0.8.0/deploy/longhorn.yaml

We can see everything created by listing all resources in the longhorn-system namespace:

kubectl get all --namespace longhorn-system

There are too many resources to list here, but wait until everything is running.

Longhorn has an admin UI that we can access by creating an ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: longhorn-system
  name: longhorn-ingress
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  rules:
  - host: longhorn-ui
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80

Originally I wanted to remove the host entry to let it bind to all hosts, and specify a more specific path, but currently there is an issue where the path must be /.

So we can leave the host and path as shown, and work around this by creating an /etc/hosts entry on our local machine to map the hostname of longhorn-ui to one of our nodes:

cat /etc/hosts
192.168.1.244 rpi-1 longhorn-ui
192.168.1.245 rpi-2
192.168.1.246 rpi-3

Thanks to Jeric Dy’s post for this idea.

In a browser, if you go to http://longhorn-ui/, you should see the dashboard like the following:

Now we can do a similar test as we did earlier to verify that everything is working…

Create a specification for a PersistentVolumeClaim and use the storageClassName of longhorn:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi

Create a specification for a Pod that binds to this PVC:

apiVersion: v1
kind: Pod
metadata:
  name: longhorn-test
spec:
  containers:
  - name: longhorn-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: longhorn-pvc
      mountPath: /data
    ports:
    - containerPort: 80
  volumes:
  - name: longhorn-pvc
    persistentVolumeClaim:
      claimName: longhorn-pvc

Create these resources with kubectl:

kubectl create -f pvc-longhorn.yaml
kubectl create -f pod-longhorn-test.yaml

After a minute or so, we should be able to see the pod running:

kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
longhorn-test 1/1 Running 0 42s 10.42.2.37 rpi-2

Get a shell to the container and create a file on the persistent volume:

kubectl exec -it longhorn-test -- sh
/ # echo "testing" > /data/test.txt

In the Longhorn UI, we should now be able to see this volume under the Volumes tab:

Drilling in to the PVC shows that the volume is replicated across all three nodes, with the data located under /var/lib/longhorn/replicas on each node:

We can’t quite as easily see the file on disk because it is stored differently, but we can inspect the location on one of the nodes to see what is there:

ssh pi@rpi-2

sudo ls -l /var/lib/longhorn/replicas/
total 4
drwx------ 2 root root 4096 May 15 17:03 pvc-317f4b66-822c-4483-ac44-84541dfac09b-bb6c2f0c

sudo ls -l /var/lib/longhorn/replicas/pvc-317f4b66-822c-4483-ac44-84541dfac09b-bb6c2f0c/
total 49828
-rw------- 1 root root 4096 May 15 17:06 revision.counter
-rw-r--r-- 1 root root 1073741824 May 15 17:06 volume-head-000.img
-rw-r--r-- 1 root root 126 May 15 17:03 volume-head-000.img.meta
-rw-r--r-- 1 root root 142 May 15 17:03 volume.meta

In this case, if our pod on node rpi-2 goes down and is relaunched on another node, it will be able to bind to the same persistent volume, since it came from Longhorn and not from host-based storage.

K3s on Raspberry Pi - Ingress

Bryan Bende — Sat, 08 May 2021 00:00:00 +0000

In this post we’ll look at how ingress works in a K3s cluster. For background, I recommend reading the Networking Section of the K3s documentation.

Ingress Overview

K3s automatically deploys the Traefik Ingress Controller and provides a service load balancer called Klipper. To see everything deployed in the kube-system namespace, run the following command:

kubectl get all --namespace kube-system

NOTE: I have my default context set to rpi-k3s so I don’t have to specify --context on every command.

This shows the following resources related to Traefik:

pod/traefik-97b44b794-dbmz2  
service/traefik
deployment.apps/traefik
replicaset.apps/traefik-97b44b794

And the following resource related to the Klipper load balancer:

pod/svclb-traefik-fc57n
pod/svclb-traefik-mj4md
pod/svclb-traefik-4qnbh
daemonset.apps/svclb-traefik

The traefik deployment contains the specification for a pod with one container using the image rancher/library-traefik:2.4.8 and having container ports 8000 (web) and 8443 (websecure).

The traefik service specifies a LoadBalancer for the traffic pod, and maps port 80 of the service to port 8000 on the traefik container, and port 443 of the service to port 8443 on the traefik container.

Klipper then creates a DaemonSet called svclb-traefik, which creates a pod on each node to act as a proxy to the service. Each of these pods is accessible from the node’s external IP address, and exposes ports 80 and 443, which map to the respective ports on the service.

The overall setup looks something like this:

Next we can deploy an example application and expose it through Traefik.

Ingress Test

This example is adapted from the Traefik documentation.

Create a namespace called whoami:

kind: Namespace
apiVersion: v1
metadata:
  name: whoami

Create a deployment for running two pods with the whoami container:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoami
  namespace: whoami
  labels:
    app: traefiklabs
    name: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: traefiklabs
      task: whoami
  template:
    metadata:
      labels:
        app: traefiklabs
        task: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami
          ports:
            - containerPort: 80

Create a service for the whoami deployment:

apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: whoami

spec:
  ports:
    - name: http
      port: 80
  selector:
    app: traefiklabs
    task: whoami

Create an ingress to link the whoami service to Traefik:

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: whoami
  namespace: whoami
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web

spec:
  rules:
    - http:
        paths:
          - path: /bar
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80
          - path: /foo
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80

With the above ingress we should now be able to access the paths /foo or /bar on port 80, using the external IP address of any node. The overall setup now looks like the following:

If we open a browser and navigate to http://192.168.1.244/bar, we get the following output:

Hostname: whoami-7d666f84d8-4fs7c
IP: 127.0.0.1
IP: ::1
IP: 10.42.1.4
IP: fe80::c414:76ff:fe4c:75cc
RemoteAddr: 10.42.2.3:39256
GET /bar HTTP/1.1
Host: 192.168.1.244
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:88.0) Gecko/20100101 Firefox/88.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Dnt: 1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.42.0.0
X-Forwarded-Host: 192.168.1.244
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-97b44b794-dbmz2
X-Real-Ip: 10.42.0.0

This shows the request reached one of the whoami containers at 10.42.1.4.

If we refresh the page, we now see the response came from the other whoami container at 10.42.2.5.

Since Traefik is performing round-robing load balancing, the requests will continue to alternate between the two whoami containers.

K3s on Raspberry Pi - Initial Setup

Bryan Bende — Fri, 07 May 2021 00:00:00 +0000

As part of trying to learn more about Kubernetes, I thought it’d be interesting to setup a mini cluster running on Raspberry Pis. I had no real purpose for doing this, but figured it would be a good learning experience and would leave me with a somewhat realistic environment.

There are already a ton of great resources that cover various aspects of running Kubernetes on Raspberry Pi. This post is just a summary of the steps that worked for me for reference.

Hardware

I decided a three node cluster would be enough to get started. Here are the items I purchased:

Most likely the 4GB model would have been more than enough, but for $20 more you can double the RAM to 8GB. Also, I already had an 8-port switch with some ethernet cables, although each Pi also has on-board Wifi as an option.

The instructions for setting up the cluster case are available online. I think it could have been made a little simpler, but overall it wasn’t too hard. Pay close attention to the orientation of everything in the pictures.

Here is what my setup looked like after putting it together:

Raspberry Pi Setup

Before we can prepare the SD cards, we have to decide which operating system to use, which then leads to thinking about which Kubernetes distribution to use.

After doing some research, it seemed like there were two main options:

Raspberry Pi OS 64-bit + K3s
Ubuntu 20.04 + MicroK8s

Since Raspberry Pi OS is the official operating system, I decided to go with that and give K3s a try.

Image SD Cards

Download the latest 64-bit version of Raspberry Pi OS
Use the Raspberry Pi Imager to image the SD cards
- Under Operating System, choose Use Custom
- Select the OS image downloaded in step 1
- Under Storage, select the SD card
- Customize settings by pressing CMD+SHIFT+X on Mac
- Set a hostname like rpi-1, enable password-based SSH
- Select Write to begin imaging
- Repeat the process for each SD card
Insert the SD cards into the Pis
Connect Pis to the switch with ethernet cables
Connect one ethernet cable from the switch to your home router
Connect a power supply to each Pi
Plug the power supplies into outlets

At this point, the Pis should boot up and get assigned a dynamic IP address on your home network, just like any other device. You can use your router’s admin console to determine the IP addresses.

Once you have the IP addresses, you can test SSH to each one by running ssh pi@<ip-address>, using the password you set when imaging the SD cards.

Assign Static IP Addresses

My router had a DHCP range of 192.168.1.2 - 192.168.1.255, so I adjusted the ending value to 192.168.1.235 in order to reserve some addresses for static assignment. That means we can pick three consecutive addresses somewhere between 192.168.1.236 - 192.168.1.255.

For this example, we’ll use:

192.168.1.244
192.168.1.245
192.168.1.246

To set the static addresses, do the following:

SSH to the first Pi
```
 ssh pi@<current-ip-address>
```
Install vim
```
 sudo apt-get install vim
```
Edit /etc/dhcpcd.conf
```
 sudo vim /etc/dhcpcd.conf
```
Find the section for static address, uncomment and edit:
```
 interface eth0
 static ip_address=192.168.1.244/24
 static routers=192.168.1.1
 static domain_name_servers=192.168.1.1
```
Replace 192.168.1.1 with your router’s IP address.

Replace 192.168.1.244 with the static address for the first Pi.
Reboot the Pi
```
 sudo reboot
```
After waiting a minute or two, SSH with the static IP
```
 ssh pi@192.168.1.244
```
Repeat the steps for each Pi.

Setup /etc/hosts mappings on local machine

 192.168.1.244 rpi-1
 192.168.1.245 rpi-2
 192.168.1.246 rpi-3

Create aliases in ~/.zshrc for easy SSH access

 alias sshrp1='ssh pi@rpi-1'
 alias sshrp2='ssh pi@rpi-2'
alias sshrp3='ssh pi@rpi-3'

At this point you can test the SSH aliases to confirm that you can SSH to each Pi using a hostname mapped to the static IP address. For more detailed information, see Setting a Static IP Address on Raspberry Pi.

Setup Password-less SSH

This is not really a requirement to run Kubernetes, but it will be more convenient to not type passwords over and over, plus it will allow us to use Ansible to issue commands to the Raspberry Pi nodes.

Generate an SSH key on your local machine
```
 ssh-keygen
```
Accept all the defaults and don’t enter a password.

This will produce ~/.ssh/id_rsa (private key) and ~/.ssh/id_rsa.pub (public key)
Copy the content of ~/.ssh/id_rsa.pub

SSH to rpi-1 and run:

 mkdir ~/.ssh
 touch ~/.ssh/authorized_keys
 chmod 0700 ~/.ssh
 chmod 0600 ~/.ssh/authorized_keys
 vi ~/.ssh/authorized_keys

Paste contents of the copied id_rsa.pub and save the file.

Repeat the process for each Pi

At this point, you should be able to run any of the SSH aliases without entering a password.

I also followed a similar process to setup password-less SSH between all of the Raspberry Pi nodes, but this is also just a nice-to-have and not necessary.

Install k3s

For installing k3s, I used the k3s-ansible setup, which required that I get Ansible on my laptop. I already had pip3 installed through homebrew, so installing Ansible amounted to:

pip3 install ansible --user
export PATH="/Users/bbende/Library/Python/3.7/bin:$PATH"

Now for running k3s-ansible…

Clone the k3s-ansible repo
Create the inventory by copying the example:
```
 cp -R inventory/sample inventory/rpi
```

Edit inventory/rpi/hosts.ini to look like the following:

 [master]
 192.168.1.244

 [node]
 192.168.1.245
 192.168.1.246

 [k3s_cluster:children]
 master
 node

Edit inventory/rpi/group_vars/all.yml to set the k3s_version:
```
 k3s_version: v1.21.0+k3s1
```

Launch the setup, this will take a few minutes:

 ansible-playbook site.yml -i inventory/rpi/hosts.ini

Configure kubectl

Assuming the setup completed successfully, you can configure kubectl on our local machine to connect to the k3s cluster.

Transfer the Kubeconfig from the master node to your local machine
```
 scp pi@192.168.1.244:~/.kube/config ~/.kube/config-rpi-k3s
```

Configure KUBECONFIG environment variable in .zshrc

 export KUBECONFIG="~/.kube/config"
 export KUBECONFIG="$KUBECONFIG:~/.kube/config-rpi-k3s"

Use kubectl to check the available config contexts

 kubectl config get-contexts

 CURRENT NAME CLUSTER AUTHINFO NAMESPACE
 * docker-desktop docker-desktop docker-desktop
           minikube minikube minikube
           rpi-k3s default default

If the name of the Raspberry Pi context is something different, you can rename it using the command:

 kubectl config rename-context <CURRENT_NAME> <NEW_NAME>

Use kubectl to view the nodes of the rpi-k3s cluster

 kubectl --context rpi-k3s get nodes

 NAME STATUS ROLES AGE VERSION
 rpi-3 Ready <none> 9d v1.21.0+k3s1
 rpi-2 Ready <none> 9d v1.21.0+k3s1
 rpi-1 Ready control-plane,master 9d v1.21.0+k3s1

At this point you should have a working k3s cluster to play around with.

As a next step, you can try following the k3s docs for installing the Kubernetes Dashboard.

Apache NiFi SAML Authentication with Keycloak

Bryan Bende — Wed, 17 Feb 2021 00:00:00 +0000

One of the features I worked on for the 1.13.0 release of NiFi was the ability to authenticate via a SAML identity provider (IDP). In this post I’ll show how you can setup NiFi to use Keycloak as the SAML IDP.

Initial NiFi Setup

In order to perform any type of authentication, we first need a secured NiFi instance. There are already many posts that cover this topic, so the starting point will be assuming that you can configure NiFi with a keystore, truststore, and https host/port.

These are the following properties that would need to be modified from the default config:

nifi.remote.input.secure=true

nifi.web.http.host=
nifi.web.http.port=

nifi.web.https.host=localhost
nifi.web.https.port=8443

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=changeit
nifi.security.keyPasswd=changeit

nifi.security.truststore=/path/to/truststore.jks
nifi.security.truststoreType=JKS
nifi.security.truststorePasswd=changeit

Keycloak Setup

Download the latest version of Keycloak and extract it somewhere. For this post I am using 12.0.2, but any recent version should be similar. After extracting, start Keycloak using the following command:

cd keycloak-12.0.2
./bin/standalone.sh -Djboss.socket.binding.port-offset=100

The port-offset is an easy way to increment the default port by 100 to avoid conflicts with other services that may already be using the default port. This makes the default port 8180, instead of 8080.

Navigate to http://localhost:8180 in your browser and follow the instructions to create an admin user for the Administration Console. After that, click the link for the Administration Console and login with the newly created user.

In the admin console, select Clients from the menu on the left, and then click the Create_button to add a new client. Enter a _Client ID (we’ll need this later when configuring NiFi), select SAML as the Client Protocol, and click Save.

There are a lot of options that can be configured, but we’ll mostly stick with defaults for now and focus on the minimal configuration to get a working setup.

Configure Root URL, Valid Redirect URIs, Base URL, and Master SAML Processing URL as follows:

Since NiFi’s SAML implementation doesn’t use a single processing URL, we also need to configure the fine-grained SAML URLs. The values for the URLs should look like the following:

We also need to tell Keycloak about the key that NiFi is going to use to sign SAML requests. So click on the SAML Keys tab, and then click Import. We are going to import from the keystore.jks that was used in nifi.properties.

We now need to create some users to authenticate with for testing. So click Users from the menu on the left and then click the Create button to add a user.

Enter user1 as the username and click Save. On the Credentials tab for user1, set a password and toggle Temporary to OFF. Repeat this same process to create another user named user2.

Next we’ll complete the NiFi SAML configuration to use Keycloak.

NiFi SAML Configuration

In nifi.properties you should see the following section of SAML related properties:

# SAML Properties #
nifi.security.user.saml.idp.metadata.url=
nifi.security.user.saml.sp.entity.id=
nifi.security.user.saml.identity.attribute.name=
nifi.security.user.saml.group.attribute.name=
nifi.security.user.saml.metadata.signing.enabled=false
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true
nifi.security.user.saml.signature.algorithm=http://www.w3.org/2001/04/xmldsig-more#rsa-sha256
nifi.security.user.saml.signature.digest.algorithm=http://www.w3.org/2001/04/xmlenc#sha256
nifi.security.user.saml.message.logging.enabled=false
nifi.security.user.saml.authentication.expiration=12 hours
nifi.security.user.saml.single.logout.enabled=false
nifi.security.user.saml.http.client.truststore.strategy=JDK
nifi.security.user.saml.http.client.connect.timeout=30 secs
nifi.security.user.saml.http.client.read.timeout=30 secs

We need to set the metadata URL to the location of Keycloak’s SAML metadata so that NiFi can retrieve this metadata during start-up. We also need to set the entity id to the value we used for the_Client ID_ when creating the SAML client in Keycloak.

nifi.security.user.saml.idp.metadata.url=http://localhost:8180/auth/realms/master/protocol/saml/descriptor
nifi.security.user.saml.sp.entity.id=org:apache:nifi:saml:sp

The last thing we need to do is configure NiFi’s authorizers.xml. We will use the file-based providers for this example, so we need to setup an initial user and initial admin that corresponds to one of the users we added to Keycloak. We’ll use user1 here.

<userGroupProvider>
    <identifier>file-user-group-provider</identifier>
    <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
    <property name="Users File">./conf/users.xml</property>
    <property name="Legacy Authorized Users File"></property>
    <property name="Initial User Identity 1">user1</property>
</userGroupProvider>

<accessPolicyProvider>
    <identifier>file-access-policy-provider</identifier>
    <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
    <property name="User Group Provider">file-user-group-provider</property>
    <property name="Authorizations File">./conf/authorizations.xml</property>
    <property name="Initial Admin Identity">user1</property>
    <property name="Legacy Authorized Users File"></property>
    <property name="Node Identity 1"></property>
    <property name="Node Group"></property>
</accessPolicyProvider>

At this point we can go ahead and start NiFi…

cd nifi-1.13.0
./bin/nifi.sh start

Open your browser and navigate to https://localhost:8443/nifi which should redirect you to the Keycloakd login page. Enter the credentials for user1 and click Sign In.

This should send you back to the NiFi UI where you are authenticated as user1.

You will notice the toolbar is disabled. This is because the current user doesn’t have any permissions to the root process group. We could create a policy and grant access to user1, but instead of giving just one user access, let’s use group-based policies.

Group Setup

In order to create group-based policies, we first need to create groups that our users can be associated with.

So click Groups from the menu on the left and then click New to create a new group. Enter group1 as the group name and click Save.

We then need to associate our user to this group. This can be done by navigating to user1, selecting the Groups tab, and joining the user to group1.

The last step is to tell the SAML client how to provide this group information in a SAML response. In order to do this we need to create a mapper in the Mappers tab of the SAML Client.

The Mapper Type should be Group List and the name can be anything you like. Leave the default attribute name of member, set the Name Format to Basic, and toggle the Full Group Path to OFF.

The result of this mapper is that each SAML response contains an attribute named member where the value is a list of the groups the user belongs to. We then have to also tell NiFi the name of this attribute by configuring the follow value:

nifi.security.user.saml.group.attribute.name=member

We’ll need to restart NiFi after making this change.

At this point, we still have the disabled toolbar, but now we can fix that. Since we are using the file-based providers for authorization, we have to tell NiFi about the existence of group1, so that we can create policies using that group.

We can add a new group by selecting Users from the menu in the top right, and then clicking the icon to add a new user/group.

We don’t need to worry about maintaining the group membership in NiFi because the SAML response is going to tell NiFi that user1 belongs to group1, so we can leave group1 empty.

Now we can create policies on the root process for “view the component” and “modify the component”. To create a policy on the root group, make sure nothing else on the canvas is selected and click the key icon in the palette on the left.

The policies should look like the following:

In order to see these policies take effect, we need to re-authenticate to NiFi via Keycloak so that NiFi gets a new SAML response with the member attribute.

So click the Logout link which will bring you to the logout landing page, and then click_Home_ which will start the login sequence again.

Since you should already be logged into Keycloak, this should immediately go back and forth between Keycloak and NiFi, and you should now see the toolbar enabled.

That’s all for this post, happy SAML’ing!

Apache NiFi Secure Cluster Setup

Bryan Bende — Tue, 23 Oct 2018 00:00:00 +0000

Setting up a secure cluster continues to generate a lot of questions, so even though several posts have already coveredthis topic, I thought I’d document the steps I performed while verifying the Apache NiFi 1.8.0 Release Candidate.

Initial Setup

The simplest cluster you can setup for local testing is a two node cluster, with embedded ZooKeeper running on the first node.

After building the source for 1.8.0-RC3, unzip the binary distribution and create two identical directories:

unzip nifi-1.8.0-bin.zip
mv nifi-1.8.0 nifi-1
cp -R nifi-1 nifi-2

The resulting directory structure should look like the following:

cluster
├── nifi-1
│   ├── LICENSE
│   ├── NOTICE
│   ├── README
│   ├── bin
│   ├── conf
│   │   ├── authorizers.xml
│   │   ├── bootstrap-notification-services.xml
│   │   ├── bootstrap.conf
│   │   ├── logback.xml
│   │   ├── login-identity-providers.xml
│   │   ├── nifi.properties
│   │   ├── state-management.xml
│   │   └── zookeeper.properties
│   ├── docs
│   └── lib
└── nifi-2
    ├── LICENSE
    ├── NOTICE
    ├── README
    ├── bin
    ├── conf
    │   ├── authorizers.xml
    │   ├── bootstrap-notification-services.xml
    │   ├── bootstrap.conf
    │   ├── logback.xml
    │   ├── login-identity-providers.xml
    │   ├── nifi.properties
    │   ├── state-management.xml
    │   └── zookeeper.properties
    ├── docs
    └── lib

Configure nifi.properties

Edit nifi-1/conf/nifi.properties and set the following properties:

nifi.state.management.embedded.zookeeper.start=true

nifi.remote.input.secure=true

nifi.web.http.port=
nifi.web.https.host=localhost
nifi.web.https.port=8443

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=yourpassword
nifi.security.keyPasswd=yourpassword
nifi.security.truststore=/path/to/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=yourpassword

nifi.cluster.protocol.is.secure=true

nifi.cluster.is.node=true
nifi.cluster.node.protocol.port=8088
nifi.cluster.flow.election.max.candidates=2

nifi.zookeeper.connect.string=localhost:2181

Edit nifi-2/conf/nifi.properties and set the following properties:

nifi.remote.input.secure=true

nifi.web.http.port=
nifi.web.https.host=localhost
nifi.web.https.port=7443

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=yourpassword
nifi.security.keyPasswd=yourpassword
nifi.security.truststore=/path/to/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=yourpassword

nifi.cluster.protocol.is.secure=true

nifi.cluster.is.node=true
nifi.cluster.node.protocol.port=7088
nifi.cluster.flow.election.max.candidates=2
nifi.cluster.load.balance.port=6343

nifi.zookeeper.connect.string=localhost:2181

NOTE: For nifi-1 I left the default value for nifi.cluster.load.balance.port and since we are running both nodes on thesame host, we need to set a different value for nifi-2.

Configure authorizers.xml

The configuration of authorizers.xml should be the same for both nodes, so edit nifi-1/conf/authorizers.xml and_nifi-2/conf/authorizers.xml_ and do the following…

Modify the userGroupProvider to declare the initial users for the initial admin and for the cluster nodes:

<userGroupProvider>
    <identifier>file-user-group-provider</identifier>
    <class>org.apache.nifi.authorization.FileUserGroupProvider</class>
    <property name="Users File">./conf/users.xml</property>
    <property name="Legacy Authorized Users File"></property>
    <property name="Initial User Identity 1">CN=bbende, OU=ApacheNiFi</property>
    <property name="Initial User Identity 2">CN=localhost, OU=NIFI</property>
</userGroupProvider>

NOTE: In this case since both nodes are running on the same host and using the same keystore, there only needs to beone user representing both nodes, but in a real setup there would be multiple node identities.

NOTE: The user identities are case-sensitive and white-space sensitive, so make sure the identities are entered exactlyas they will come across to NiFi when making a request.

Modify the accessPolicyProvider to declare which user is the initial admin and which are the node identities:

<accessPolicyProvider>
    <identifier>file-access-policy-provider</identifier>
    <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
    <property name="User Group Provider">file-user-group-provider</property>
    <property name="Authorizations File">./conf/authorizations.xml</property>
    <property name="Initial Admin Identity">CN=bbende, OU=ApacheNiFi</property>
    <property name="Legacy Authorized Users File"></property>
    <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
    <property name="Node Group"></property>
</accessPolicyProvider>

Configure state-management.xml

The configuration of state-management.xml should be the same for both nodes, so edit nifi-1/conf/state-management.xml and_nifi-2/conf/state-management.xml_ and do the following…

Modify the cluster state provider to specify the ZooKeeper connect string:

<cluster-provider>
    <id>zk-provider</id>
    <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
    <property name="Connect String">localhost:2181</property>
    <property name="Root Node">/nifi</property>
    <property name="Session Timeout">10 seconds</property>
    <property name="Access Control">Open</property>
</cluster-provider>

Configure zookeeper.properties

The configuration of zookeeper.properties should be the same for both nodes, so edit nifi-1/conf/zookeeper.properties and_nifi-2/conf/zookeeper.properties_ and do the following…

Specify the servers that are part of the ZooKeeper ensamble:

server.1=localhost:2888:3888

Create ZooKeeper myid file

Since embedded ZooKeeper will only be started on the first node, we only need to do the following for nifi-1:

mkdir nifi-1/state
mkdir nifi-1/state/zookeeper
echo 1 > nifi-1/state/zookeeper/myid

Start Cluster

./nifi-1/bin/nifi.sh start && ./nifi-2/bin/nifi.sh start

At this point, assuming you have the client cert (p12) for the initial admin in your browser (“CN=bbende, OU=ApacheNiFi”)then you can access your cluster at https://localhost:8443/nifi or https://localhost:7443/nifi.

Common Issues

Embedded ZooKeeper fails to create directory

 java.io.IOException: Unable to create data directory ./state/zookeeper/version-2
     at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:85)
     at org.apache.nifi.controller.state.server.ZooKeeperStateServer.startStandalone(ZooKeeperStateServer.java:85)
     ... 51 common frames omitted

This happens frequently with embedded ZooKeeper and usually you can just try to start again and it will resolve itself.

Mistake in initial admin identity, or node identity

If you made a typo in any of the identities and need to edit authorizers.xml, you MUST delete conf/users.xml and conf/authorizations.xml from each node before restarting.

The reason is that the initial admin and node identities are only used when NiFi starts with no other users, groups, or policies. If you don’t delete the users and authorization files, then your changes will be ignored.

Apache NiFi Registry 0.2.0

Bryan Bende — Wed, 20 Jun 2018 00:00:00 +0000

The Apache NiFi community recently completed the second release of NiFi Registry (0.2.0). This post will introduce someof the new features, including git-based flow persistence, a more configurable metadata database, and event hooks.

Background

Before jumping into the new features, lets review how NiFi Registry works behind the scenes.

The user interface is a single page webapp built with Angular, and communicates with the server via the NiFi RegistryREST API. Behind the REST API is a service layer where the primary business logic is implemented, and the servicelayer interacts with the metadata database and flow persistence provider.

The metadata database stores information about buckets and versioned items, such as identifiers, names, descriptions,and commit comments, as well as which items belong to which bucket. The initial release utilized an embedded H2 databasethat was primarily hidden from end users, except for configuring the directory location.

The flow persistence provider stores the content of each versioned flow and is considered a public extensionpoint. This means custom implementations can be provided by implementing the FlowPersistenceProvider interface. Theinitial release provided a file-system implementation of flow persistence provider which used the local file-systemfor persistence.

The overall idea is that the metadata database contains information across all types of versioned items (which may eventuallybe more than just flows), and each type of item may have its own persistence mechanism for the content of the item.

Git FlowPersistenceProvider

The 0.2.0 release provides a new git-based implementation of the FlowPersistenceProvider utilizing the JGit library.This means the content of versioned flows can now be stored in a git repository.

The git provider can be configured in providers.xml via the following configuration:

<flowPersistenceProvider>
    <class>
      org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider
    </class>
    <property name="Flow Storage Directory">./flow_storage</property>
    <property name="Remote To Push"></property>
    <property name="Remote Access User"></property>
    <property name="Remote Access Password"></property>
</flowPersistenceProvider>

The “Flow Storage Directory” property specifies a local directory that is expected to already be a git repository. Thiscould be done by creating a new directory and running “git init”, or from cloning an existing remote repository.

The “Remote To Push” property specifies the name of remote to automatically push to. This property is optional and if notspecified then commits will remain in the local repository, unless a push is performed manually.

NOTE: In order to use GitHub, do the following:

Create a new GitHub repo and clone it locally
Set “Flow Storage Directory” to the directory where the repo was cloned
Go to GitHub’s “Developer Settings” for your account
Create a new “Personal Access Token” for your NiFi Registry instance
Set “Remote Access User” to your GitHub username
Set “Remote Access Password” to the access token

External Metadata Database

The 0.2.0 release provides new configuration that allows the metadata database to utilize an external database.Currently Postgres is the only supported database besides H2, although others may work with additional testing.

The 0.1.0 release originally had two properties in nifi-registry.properties related to the H2 database:

nifi.registry.db.directory=
nifi.registry.db.url.append=

In order to make the database more configurable, the 0.2.0 release adds the following properties:

nifi.registry.db.url=
nifi.registry.db.driver.class=
nifi.registry.db.driver.directory=
nifi.registry.db.username=
nifi.registry.db.password=

If you are starting NiFi Registry for the first time, then the properties will default to use H2 and you don’t need todo anything unless you want to change the configuration to use a Postgres database.

If you are upgrading an existing NiFi Registry, it will determine if the old nifi.registry.db.directory property ispopulated and determine if the directory contains an existing H2 database. If an existing database is found, and ifthe target database is empty, then data will automatically be migrated from the existing database to the new database.

NOTE: This essentially offers a one-time migration from the existing H2 database, to a new H2 database, or a new Postgresdatabase.

For specifics about each property please refer to the NiFi Registry Administration Guide.

Event Hooks

An event hook is a new extension point that allows custom code to be triggered when application events occur.

In order to implement an event hook, the following Java interface must be implemented:

public interface EventHookProvider extends Provider {

    void handle(Event event) throws EventHookException;

}

The Event object will contain an EventType, along with a list of field/value pairs that are specific to the event. At thetime of writing this, the possible event types are the following:

CREATE_BUCKET
CREATE_FLOW
CREATE_FLOW_VERSION
UPDATE_BUCKET
UPDATE_FLOW
DELETE_BUCKET
DELETE_FLOW

The list of event types and fields can be found in the code in the EventType class.

The 0.2.0 release comes with two provided event hooks, LoggingEventHookProvider and ScriptedEventHookProvider.

LoggingEventHookProvider

The LoggingEventHookProvider logs a string representation of each event using an SLF4J logger. The logger can beconfigured via NiFi Registry’s logback.xml, which by default contains an appender that writes to a log file namednifi-registry-event.log in the logs directory.

To enable this hook, simply uncomment the configuration in providers.xml:

<eventHookProvider>
    <class>
      org.apache.nifi.registry.provider.hook.LoggingEventHookProvider
    </class>
</eventHookProvider>

After creating a bucket and starting version control on a process group, the event log should show something like the following:

2018-06-19 11:30:46,174 ## CREATE_BUCKET [BUCKET_ID=5d81dc5e-79e1-4387-8022-79e505f5e3a0, USER=anonymous]
2018-06-19 11:32:16,503 ## CREATE_FLOW [BUCKET_ID=5d81dc5e-79e1-4387-8022-79e505f5e3a0, FLOW_ID=a89bf6b7-41f9-4a96-86d4-0aeb3c3c25be, USER=anonymous]
2018-06-19 11:32:16,610 ## CREATE_FLOW_VERSION [BUCKET_ID=5d81dc5e-79e1-4387-8022-79e505f5e3a0, FLOW_ID=a89bf6b7-41f9-4a96-86d4-0aeb3c3c25be, VERSION=1, USER=anonymous, COMMENT=v1]

ScriptEventHookProvider

The ScriptedEventHookProvider allows a custom script to be executed for each event. This can be used to handle manysituations without having to formally implement an event hook in Java.

The configuration for this event hook looks like the following:

<eventHookProvider>
    <class>
      org.apache.nifi.registry.provider.hook.ScriptEventHookProvider
    </class>
    <property name="Script Path"></property>
    <property name="Working Directory"></property>
</eventHookProvider>

The “Script Path” property is the full path to a script that will executed for each event. The arguments to the scriptwill be the event fields in the order they are specified for the given event type. As an example, lets say we created ascript called logging-hook.sh with the following content:

#!/bin/bash
echo $@ >> /tmp/nifi-registry-event.log

For each event, this would echo the arguments to the script and append them to /tmp/nifi-registry-event.log. After creatinga bucket and starting version control on a process group, the log should show something like the following:

CREATE_BUCKET feeb0fbe-5d7e-4363-b58b-142fa80775e1 anonymous
CREATE_FLOW feeb0fbe-5d7e-4363-b58b-142fa80775e1 1a0b614c-3d0f-471a-b6b1-645e6091596d anonymous
CREATE_FLOW_VERSION feeb0fbe-5d7e-4363-b58b-142fa80775e1 1a0b614c-3d0f-471a-b6b1-645e6091596d 1 anonymous v1

Summary

The new features in the 0.2.0 release provide additional options for the metadata database and flow persistence provider,allowing data to be pushed off the server where the application is running, and on to external locations. In addition,event hooks provide an easy way to initiate custom workflows based on various events.

Apache NiFi - Secure Keytab Access

Bryan Bende — Mon, 09 Apr 2018 00:00:00 +0000

In a multi-tenant environment, a typical approach is to create a process group per team and apply securitypolicies that restrict access, such that a given team only has access to it’s own group.

Within a process group, a team may need to interact with an external system that is secured with Kerberos. As anexample, lets say there is a Kerberized HDFS cluster, and there are two teams that each have a keytab toaccess this cluster.

In order to create a PutHDFS processor that sends data to the Kerberized HDFS cluster, the processor must be configuredwith a principal and keyab, and the keytab must be on a filesystem that is accessible to the NiFi JVM. In addition, the keytab must be readable by the operating system user that launched the NiFi JVM.

The problem here is that the Keytab property is a free-form text field, so there is no way to stop team 1 from entering team 2’s keytab.

In order to solve this problem, the 1.6.0 release introduced a Keytab Controller Service, along with more granular restricted components, which together can be used to properly secure access to keytabs.

Keytab Controller Service

The first step to securing keytab access was to create a controller service where the principal and keytab could bedefined, and then update all processors that had the free-form properties to also have a property for a KeytabService.

NOTE: The free-form properties were not removed on this release in order to give everyone a chance to migrate tothe new approach, but we’ll discuss this more later.

The benefit of using a controller service is that we can restrict which users have ability to use the service via security policies. In our example above, an admin user can create a Keytab Service for team 1 that points to team 1’s keytab, and then create a policy that gives only team 1 access to this service. The admin user can then do the same thing for team 2.

This gets us very close to what we want, BUT what if someone from team 1 ignores the service that was created for them, and just creates another Keytab Service pointing to team 2’s keytab?

We need a way to also restrict who can create Keytab Controller services, so that only the admin users can create them,and not any of the users in team 1 or team 2.

Granular Restricted Components

The NiFi 1.0.0 release introduced the concept of “restricted components”. The idea was that some components may beable to do things that we don’t want all users to do, such as access the local file system, or execute code. These types of components required a user to have an additional “restricted” permission in order to create the component.

The 1.6.0 release introduced more granular categories of restricted components:

read-filesystem
write-filesystem
execute-code
access-keytab
export-nifi-details

The restricted permissions are controlled from the global policies menu in the top-right:

In our previous example, if we only give the admin users the “access-keytab” permission, then the users in team 1 and team 2 won’t be able to create a Keytab Controller service, thus forcing them to use the Keytab Services they were given permission to.

Preventing Free-form Keytab Properties

The final piece to the puzzle is to prevent the use of the old free-form keytab properties that were left around forbackwards compatibility. This can be done by configuring an environment variable in nifi-env.sh:

export NIFI_ALLOW_EXPLICIT_KEYTAB=true

Setting this value to false will produce a validation error in any component where the free-form keytab property isentered, which means the component can’t be started unless it uses a Keytab Controller service.

Summary

In summary…

Only admin users have the “access-keytab” permission
Admin users create Keytab Services and assign appropriate permissions
Regular users can use the Keytab Services they were given access to, but can’t create new ones
Legacy free-form properties are disallowed via the NIFI_ALLOW_EXPLICIT_KEYTAB environment variable