DEV Community: Anthony Ikeda

Getting started with Spring AI

Anthony Ikeda — Fri, 29 Mar 2024 21:28:10 +0000

I've been trying to get started on this sample ever since they announced SpringAI a few weeks ago and to be honest I wasn't sure what to expect. But having gotten my hands dirty with this first PoC I have to say I'm impressed.

Before we begin, the context of this post is about integrating AI services (OpenAI, Gemini, HuggingFace etc) into an existing application.

This is not about training models, nor is it about creating capabilities that generate ML offerings, purely integrating with the existing offerings out there.

With that, I'll begin...

AI over the years

I've been watching AI for the past 20 odd years and of course it comes in waves of new achievements but for some reason it never really became mainstream until recently. We now see this tidal wave of offerings that offer services from chat, image generation, Transcription, even pair programming so I will say we are seeing a more viable partnership with AI in the way we write and design applications.

Just about every cloud provider and startup is now creating capabilities that enable people to tailor how to incorporate AI into their own products to create a new approach to how you interact with their services.

Spring AI

Spring AI was announced a few weeks ago, but what exactly is it?

Spring AI is the framework that allows you to easily select an AI provider and seamlessly integrate it with your existing application stack.

Spring AI takes Spring Boot and incorporates the right interfaces to ensure you don't have to rewrite code each time you decide to switch AI providers.

Let's get started!

The Spring AI documentation has a brief example of how to get started, so we will make use of that and explore beyond what it offers.

Bootstrap Spring

As per the article, we will use Spring CLI (NOT TO BE CONFUSED WITH SPRING BOOT CLI!!!!) to get up and running.

$ spring boot new --from ai --name myai
Getting project from https://github.com/rd-1-2022/ai-openai-helloworld
Created project in directory 'myai'

The first you notice about the application is that it is a typical Spring Boot application. It comes with your typical Application.java entry point and also includes a RestController SimpleAiController.java

SimpleAiController.java

What I like about Spring AI is its simplicity. Let's take a quick look at the RestController it creates.

package org.springframework.ai.openai.samples.helloworld.simple;

import org.springframework.ai.chat.ChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.util.Map;

@RestController
public class SimpleAiController {

    private final ChatClient chatClient;

    @Autowired
    public SimpleAiController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @GetMapping("/ai/simple")
    public Map<String, String> completion(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
        return Map.of("generation", chatClient.call(message));
    }
}

As you can see, the most obvious aspect is the ChatClient. This is the interface that makes the calls to the provider, returning the results from your prompt.

If you look at the pom.xml you'll notice the only change from a standard SpringBoot application is the AI integration:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>

This is the implementation specific classes that perform the integration with the OpenAI platform.

In order to use this API there are still a couple of steps required.

Up and Running

First you need to create an OpenAI account and add a balance.

Once you've signed up, navigate to Settings -> Billing -> Payment Methods and add a method of payment.

Next, go back to Settings -> Billing and add a small value, $5 is usually enough to start playing around with the LLMs.

Lastly, we are going to create an API key to allow our Spring AI application to execute requests. Navigate to API Keys and select + Create new secret key

In the dialog that is presented, give your key a name, select All permissions and click the Create secret key button.

You will then be presented with your secret key which you will use for your Spring AI application - copy it and put it somewhere safe.

Configuring our Chat API

In the application.properties file we will add 2 properties:

spring.ai.openai.api-key=<secret api key>
spring.ai.openai.chat.options.model=gpt-3.5-turbo

This is where we enable our application to:

perform requests against our OpenAI account
Make use of the gpt-3.5-turbo chat model

OpenAI offers a number of different models for various purposes, this is one of the flexible features around Spring Ai that makes it easy to integrate AI into your applications.

The OpenAI Model documentation lists the models that are available for use.

A list of the model prices is also available here

With these settings in place, you can start your application and begin sending prompts!

$ ./mvnw spring-boot:run
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.3)

2024-03-29T14:04:33.137-07:00  INFO 46846 --- [           main] o.s.a.o.samples.helloworld.Application   : Starting Application using Java 21.0.2 with PID 46846 (/Users/anthonyikeda/work/git/myai/target/classes started by anthonyikeda in /Users/anthonyikeda/work/git/myai)
2024-03-29T14:04:33.139-07:00  INFO 46846 --- [           main] o.s.a.o.samples.helloworld.Application   : No active profile set, falling back to 1 default profile: "default"
2024-03-29T14:04:34.316-07:00  INFO 46846 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-03-29T14:04:34.328-07:00  INFO 46846 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-03-29T14:04:34.328-07:00  INFO 46846 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.19]
2024-03-29T14:04:34.521-07:00  INFO 46846 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-03-29T14:04:34.522-07:00  INFO 46846 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1344 ms
2024-03-29T14:04:35.260-07:00  INFO 46846 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-03-29T14:04:35.360-07:00  INFO 46846 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path ''
2024-03-29T14:04:35.373-07:00  INFO 46846 --- [           main] o.s.a.o.samples.helloworld.Application   : Started Application in 2.587 seconds (process running for 3.021)

Just like any Spring Boot application you can see the application started on port 8080.

Running your favorite HTTP Client, let's send some prompts.

GET http://localhost:8080/ai/simple?message=Who%20was%20first%20to%20live%20in%20Pangea
Accept: application/json

The first organisms to live on Pangea were likely single-celled bacteria and algae that existed around 300 million years ago. As Pangea was a supercontinent that formed during the late Paleozoic era, it provided a unique environment for early life forms to evolve and thrive.

Success!

I agree, what we have created is a proxy or gateway to these services, but with minimal modifications, getting started too less than 30 minutes!

Functional Kafka with Spring Cloud - Part 3

Anthony Ikeda — Wed, 27 Apr 2022 22:25:45 +0000

So, we have messages going to Kafka and messages coming off Kafka. Right now, though, those messages are rather unstructured.

There are a couple of approaches we can take:

Serialize an object using JSON
Serialize an object using Avro

Well, we've used JSON a lot in the past for our APIs and to be honest, it's boring hehe. So in this article we will take advantage of Apache Avro and create binary representations of the messages in Kafka!

Prerequisites:

We will need our Employee API from the first article and the Consumer from the second article.
Schema Registry which we downloaded and configured in the first article but didn't touch quite just yet!

First let's update the Employee API and set up Avro!

Add Avro to the Employee API

We will need a new dependency this time but first we need to reference a new maven repository. Add the following tag under the pom.xml project tag:

<repositories>
    <repository>
        <id>confluent</id>
        <url>https://packages.confluent.io/maven/</url>
    </repository>
</repositories>

This will make the Confluent Maven repo available from which we will source the Avro Serializers.

Next, let's add the the new dependency:

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-avro-serializer</artifactId>
    <version>7.1.1</version>
</dependency>

As mentioned, this dependency supplies us with the tools to convert our Java objects to and from Avro. To complete this tasks we need one more change to our pom.xml:

<build>
    <plugins>
...
        <plugin>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <version>1.11.0</version>
            <executions>
                <execution>
                    <id>schemas</id>
                    <phase>generate-sources</phase>
                    <goals>
                        <goal>schema</goal>
                        <goal>protocol</goal>
                        <goal>idl-protocol</goal>
                    </goals>
                    <configuration>
                        <sourceDirectory>${project.basedir}/src/main/resources/</sourceDirectory>
                        <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The avro-maven-plugin performs some nifty functions for us, primarily it will take an Avro schema and generate the classes we need to convert our classes to and from Avro.

As you can see, we've configured it to read the JSON schemas from the ${project.basedir}/src/main/resources/ directory and publish the generated classes to ${project.basedir}/src/main/java/

Now we can define our new Employee.java class.

Create a new file src/main/resources/employee.avsc:

{
  "type" : "record",
  "name" : "Employee",
  "namespace": "com.example.demo.model",
  "fields": [
    {
      "name" : "employeeId",
      "type" : "string"
    },
    {
      "name": "firstname",
      "type": "string"
    },
    {
      "name": "lastname",
      "type": "string"
    }]
}

Here we have defined the structure of data we want to transport. There are 3 areas worth noting:

"name" : "Employee"

This is the name of the record as an entity. It's how we will refer to the record and it will end up being the name of the class that is generated.

"namespace" : "com.example.demo.model"

The namespace is synonymous with a Java package, it helps to organize the Records and allow us to create other classes of the same name in different namespaces.

"fields" : [...]

Here we have the structure of the message, basically the record attributes.

When you build the application with:

$ ./mvnw clean verify

You will have a new class generated in the src/main/java directory: com.example.demo.model.Employee

It's too large a class to publish here, however, it contains logic to identify the fields, field types and the encoders/decoders from and to the binary representation.

Now that we have the Avro entity created, we want to update the code to publish this to kafka.

Update the EmployeeController

First change we want is to update the KafkaTemplate<String, String> to KafkaTemplate<String, Employee>:

@RequestMapping("/employee")
@RestController
public class EmployeeController {

    @Autowired
    protected KafkaTemplate<String, Employee>  kafka;

This ensures the message ID remains as a String value, but the Employee object get's recognized as a serialized/deserialized object.

Next let's change how we send the message in our @PostMapping:

    @PostMapping
    public ResponseEntity<Void> createEmployee(@RequestParam("firstname") String firstname,
                                               @RequestParam("lastname") String lastname) {

        String id = UUID.randomUUID().toString();
        Employee employee = new Employee(id, firstname, lastname);
        kafka.send("employee", id, employee);

        return ResponseEntity.created(URI.create(String.format("/employee/%s", id))).build();
    }

The only thing we've changed here is the setting up of the Employee record and how we send it to the KafkaTemplate.

There is one last change to make and that is to the src/main/resources/application.yml file.

We need to tell the application to handle the Record Value differently, to treat is as an Avro record. So we need to change the value serializer/deserializer:

spring:
  kafka:
    properties:
      schema.registry.url: http://localhost:8081
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
    bootstrap-servers:
      - localhost:9092

server:
  port: 8070

As mentioned before, we want to preserve the message identifier as a string, but take the value and serialize it using Avro.

You may also notice a new entry:

spring:
  kafka:
    properties:
      schema.registry.url: http://localhost:8081

This is the Schema Registry we downloaded in the first article that is going to store the schema of our Employee record so other consumers can perform the same transformation in absence of the actual schema (employee.avsc)

With that, let's start the Schema Registry:

$ $CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties

As the schema registry starts up, it will create it's own topic in Kafka where it stores the schemas.

Now we can start publishing to Kafka using Avro!

Start the Employee and send a call to the POST endpoint.

We really shouldn't see any difference from the application standpoint, but if you inspect the topic employee using the kafka-console-consumer you should see the message in a very different format:

❯ bin/kafka-console-consumer.sh --topic employee --bootstrap-server localhost:9092 --from-beginning
H0e426803-0903-43dd-a63a-a66734c52b0e
                                     AndrewMarshall

Not much to see really, but this is our Avro representation of the message.

If you try and consume this with the current application consumer we created it won't really be able to read the data properly. So let's fix that!

Update the Kafka Consumer

In our second application these are the changes we need to make:

Create a representation class for the Employee record
Add the Avro serializers to the project
Reconfigure the Serializers for the Record Value (not the message id)
Reference the Employee attributes instead of printing a raw string to the log

Employee.java

We want to mirror the original Avro record by creating a simple class with the same namespace:

package com.example.demo.model;

public class Employee {

    private String employeeId;

    private String firstname;

    private String lastname;

    public Employee() {
    }

    public Employee(String employeeId, String firstname, String lastname) {
        this.employeeId = employeeId;
        this.firstname = firstname;
        this.lastname = lastname;
    }

    public String getEmployeeId() {
        return employeeId;
    }

    public void setEmployeeId(String employeeId) {
        this.employeeId = employeeId;
    }

    public String getFirstname() {
        return firstname;
    }

    public void setFirstname(String firstname) {
        this.firstname = firstname;
    }

    public String getLastname() {
        return lastname;
    }

    public void setLastname(String lastname) {
        this.lastname = lastname;
    }
}

Nothing crazy here, just a standard Plain Old Java Object.

Add Avro Serializers to the project

Much like we did with the API we want to add Avro to our project, the only difference is we won't be using the plugin.

In your pom.xml add the following entries:

<dependencies>
...
    <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-avro-serializer</artifactId>
        <version>7.1.1</version>
    </dependency>
...
</dependencies>

<repositories>
    <repository>
        <id>confluent</id>
        <url>https://packages.confluent.io/maven/</url>
    </repository>
</repositories>

Reconfigure the Serializers for the Record Value

Next let's change the src/main/resources/application.yml file:

spring:
  cloud:
    stream:
      kafka:
        binder:
          producer-properties:
            value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
            key.serializer: org.apache.kafka.common.serialization.StringSerializer
            schema.registry.url: http://localhost:8081
          consumer-properties:
            value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
            key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
            schema.registry.url: http://localhost:8081
            value.deserializer.specific.avro.reader: true

Here we have replaced the value serializers/deserializers with the io.confluent.kafka.serializers classes.

One other thing to note is we specified the Schema Registry to locate the schemas and we've also specified that when it comes to the record values, we want to manage them specifically with Avro:

spring:
  cloud:
    stream:
      kafka:
        binder:
          consumer-properties:
            value.deserializer.specific.avro.reader: true

Reference the `Employee` attributes instead of printing a raw string to the log

Here we will update our consumer code to use the Employee class instead of a string:

@Bean
public Function<Flux<Employee>, Flux<Employee>> employeeProcessor() {
    return employee -> employee.map(emp -> {
       log.info("Employee is {}, {}, {}", emp.getEmployeeId(), emp.getFirstname(), emp.getLastname());
       return emp;
    }).log();
}

@Bean
public Consumer<Employee> employeeConsumer() {
    return (value) -> log.info("Consumed {}", value.getEmployeeId());
}

All good, we should now be ready.

Before we start the application we need to do 2 things:

Start the Schema Registry
Clear the current topics

Start the Schema Registry

Let's start it up with:

$ $CONFLUENCE_HOME/bin/schema-registry-start ./etc/schema-registry/schema-registry.properties

In kafka, for now, we will just delete the topics we were using to recreate them (make sure neither of the Spring apps are running):

$ $KAFKA_HOME/bin/kafka-topics.sh --delete  --bootstrap-server=localhost:9092 --topic employees-processed
$ $KAFKA_HOME/bin/kafka-topics.sh --delete --bootstrap-server=localhost:9092 --topic employees

Start up the Employee API and the Consumer and once you POST to the Employee API you should see the Consumers reading the message and printing out the attributes!

Functional Kafka with Spring Cloud - Part 2

Anthony Ikeda — Wed, 27 Apr 2022 21:04:24 +0000

So, previously we setup a basic kafka environment and a Spring Boot Web API to push a basic string message to Kafka, let's spend the time in this article reading the value.

Continuing on from where we left off, we need to create another Spring Application using the https://start.spring.io website:

You can omit the Spring Web dependency this time around!

We aren't going to add anything fancy yet, just get the code base and get it ready to start a new application!

Set Up Spring Cloud

First, let's set up Spring Cloud dependencies. In your pom.xml let' make the following changes:

Add a new property indicating the version for Spring Cloud:

<properties>
    <java.version>18</java.version>
    <spring-cloud-release.version>2021.0.1</spring-cloud-release.version>
</properties>

Next we want to reference the Spring Cloud POM. Add the following XML block under the project tag:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-dependencies</artifactId>
            <version>${spring-cloud-release.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

This will include any dependencies we need from Spring Cloud v2021.0.1

For our String based consumer we just need 2 core dependencies added and 1 test dependency:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream</artifactId>
    <scope>test</scope>
    <classifier>test-binder</classifier>
    <type>test-jar</type>
</dependency>

Consumers and Processors

Now we can add in the code that will basically be a data pipeline, taking a message from a topic and then doing something with it before passing it on to another topic.

Create a new class KafkaConfiguration.java with the following beans:

package com.example.demo;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import reactor.core.publisher.Flux;

import java.util.function.Consumer;
import java.util.function.Function;


@Configuration
public class KafkaConfiguration {

    private static final Logger log = LoggerFactory.getLogger(KafkaConfiguration.class);

    @Bean
    public Function<Flux<String>, Flux<String>> employeeProcessor() {
        return employee -> employee.map(emp -> {
           log.info("Employee is {}", emp);
           return emp;
        }).log();
    }

    @Bean
    public Consumer<String> employeeConsumer() {
        return (value) -> log.info("Consumed {}", value);
    }

}

Let's walk the code!

We have a basic Logger in place to output values as they happen:

private static final Logger log = LoggerFactory.getLogger(KafkaConfiguration.class);

Next we have a bean that will act as an Input/Output of the message coming from the topic:

@Bean
public Function<Flux<String>, Flux<String>> employeeProcessor() {
    return employee -> employee.map(emp -> {
       log.info("Employee is {}", emp);
       return emp;
    }).log();
}

Our method signature represents a (data in/data out) pipeline:

Function<Flux<String>, Flux<String>>

Here we are basically saying that the message we read in will be a string (<Flux<String>,)and we will then send a message that is also a string (, Flux<String>>)

The body of our processor just reads the message and prints it to the log (a couple of times!):

return employee -> employee.map(emp -> {
       log.info("Employee is {}", emp);
       return emp;
    }).log();

Typically we might call another method to do more with the incoming data creating perhaps a different output. For now we just send the message as-is on it's way.

The next bean is a consumer, for now it just logs out the String message:

@Bean
public Consumer<String> employeeConsumer() {
    return (value) -> log.info("Consumed {}", value);
}

Before we can run this application you may notice something missing, the topics aren't yet defined.

When we use Spring Cloud Functions to create these messaging apps, we need to defer to the binding capabilities whereby we configure these beans and bind them to the underlying topics of the message bus.

Open the src/main/resources/application.yml file and let's do that now!

spring:
  cloud:
    function:
      definition: employeeConsumer;employeeProcessor
    stream:
      bindings:
        employeeProcessor-in-0:
          destination: employee
        employeeProcessor-out-0:
          destination: employees-processed
        employeeConsumer-in-0:
          destination: employees-processed

      kafka:
        binder:
          producer-properties:
            value.serializer: org.apache.kafka.common.serialization.StringSerializer
            key.serializer: org.apache.kafka.common.serialization.StringSerializer
          consumer-properties:
            value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
            key.deserializer: org.apache.kafka.common.serialization.StringDeserializer

First in our config, we've created function definitions:

spring:
  cloud:
    function:
      definition: employeeConsumer;employeeProcessor

As you can see these are simply the names of the beans we just defined. We will use these as reference points later on in the configuration.

Next we've defined the actual Bean-to-Topic bindings:

spring:
  cloud:
    stream:
      bindings:
        employeeProcessor-in-0:
          destination: employee
        employeeProcessor-out-0:
          destination: employees-processed
        employeeConsumer-in-0:
          destination: employees-processed

Using the definitions (employeeProcessor, employeeConsumer) we have mapped the topics accordingly. So, the employeeProcessor has 2 mappings:

Message In: employee
Message Out: employees-processed

Meanwhile the employeeConsumer has 1 binding:

Message In: employees-processed

This means that anything that lands on the employee topic, gets picked up by the employeeProcesor. The employeeProcessor then prints out the message then sends the message to the employees-processed topic.

Any messages coming to the employees-processed queue then get picked up by the employeeConsumer which, again, just prints the message out to the logs.

Lastly in our configuration we are defining some basic properties for Kafka in general:

spring:
  cloud:
    stream:
      kafka:
        binder:
          producer-properties:
            value.serializer: org.apache.kafka.common.serialization.StringSerializer
            key.serializer: org.apache.kafka.common.serialization.StringSerializer
          consumer-properties:
            value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
            key.deserializer: org.apache.kafka.common.serialization.StringDeserializer

Quite simply put, we treat messages coming off and going onto the topics as Strings!

Starting the application

You now be able to run the application with:

$ ./mvnw spring-boot:run

Provided the original web application from the previous article is still running, we should be able to send a message from the API and see it being picked up by our consumer:

Employee is Marshall, Andrew
Consumed Marshall, Andrew

In the next article we will introduce Avro as the serializer and show you how to send binary messages and the Schema Registry!

Functional Kafka with Spring Cloud - Part 1

Anthony Ikeda — Wed, 27 Apr 2022 20:30:46 +0000

I've put this article together to create a working demo of Spring Cloud Kafka which I've been unable to find so far.

Prerequisites:

Java 11+ (I'm using Java 18)
Linux based OS (I'm on macOS)
Httpie

Setup:

Spring Cloud 2021.0.1
Confluent Schema Registry 7.1.0
Apache Kafka 2.13_3.1.0
Apache ZooKeeper 3.7.0

This article will first start with setting up a web API publishing events to Kafka as a string with a functional kafka consumer using Spring Cloud Stream.

Set up the environment

Download Apache ZooKeeper from here:

https://zookeeper.apache.org/releases.html#download

Decompress it and move it to a working folder (we will use $ZOOKEEPER_HOME)

Download Kafka from here:

https://kafka.apache.org/downloads

Again, decompress the archive and move it to a working folder, this time we will refer to the working folder as $KAFKA_HOME

Download and decompress the Schema Registry using the following:

$ wget https://packages.confluent.io/archive/7.0/confluent-community-7.0.1.tar.gz
$ tar -xf confluent-community-7.0.1.tar.gz
$ cd confluent-7.0.1

We will refer to this applications location at $CONFLUENT_HOME

Configure the environment

First let's set up ZooKeeper!

ZooKeeper Config

Create the config file by copying the example config:

ZOOKEEPER_HOME $> cp conf/zoo_sample.cfg conf/zoo.cfg

The only value you may want to edit is the dataDir directory to a location that you are comfortable storing the metadata to:

...
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=./data

Save the file and we will move on to configuring Kafka.

Kafka Config

The file we are interested here is:

$KAFKA_HOME/config/server.properties

There are 2 values we most likely will change:

log.dirs=./logs
zookeeper.connect=localhost:2181

This ensures our logs are in an accessible location as well as allows Kafka to connect to ZooKeeper to persist data.

Configure Schema Registry

Just one file needs to be checked here: $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties

All the default should be fine:

listeners=http://0.0.0.0:8081
kafkastore.bootstrap.servers=PLAINTEXT://localhost:9092
kafkastore.topic=_schemas
debug=false

Starting ZooKeeper and Kafka

We won't be using the Schema Registry to begin with since we are just working with String values in Kafka so let's get them started:

$ZOOKEEPER_HOME > bin/zkServer.sh start conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Starting zookeeper ... STARTED

$KAFKA_HOME > bin/kafka-server-start.sh config/server.properties
...
[2022-04-25 11:19:17,742] INFO Kafka version: 3.1.0 (org.apache.kafka.common.utils.AppInfoParser)
[2022-04-25 11:19:17,743] INFO Kafka commitId: 37edeed0777bacb3 (org.apache.kafka.common.utils.AppInfoParser)
[2022-04-25 11:19:17,743] INFO Kafka startTimeMs: 1650910757737 (org.apache.kafka.common.utils.AppInfoParser)
[2022-04-25 11:19:17,745] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Okay, let's get programming!!!

Employee API

This will be a simple API that will havea single POST endpoint to create a kafka message.

Just head to https://start.spring.io and create a basic web application:

Click download and expand the demo.zip file to a workspace

To get started we are going to need some dependencies to publish to Kafka.

To your pom.xml file add the following dependency:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.8.4</version>
</dependency>

Next we will create the controller to serve our endpoint.

package com.example.demo;

import com.example.demo.model.Employee;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.net.URI;
import java.util.UUID;

@RequestMapping("/employee")
@RestController
public class EmployeeController {

    @Autowired
    protected KafkaTemplate<String, String>  kafka;

    public ResponseEntity<Void> createEmployee(@RequestParam("firstname") String firstname,
                                               @RequestParam("lastname") String lastname) {

        String id = UUID.randomUUID().toString();
        kafka.send("employee", id, String.format("%s, %s", lastname, firstname));

        return ResponseEntity.created(URI.create(String.format("/employee/%s", id))).build();
    }

}

Let's quickly walk through the code...

First, we inject the KafkaTemplate<String, String> template since we aren't doing anything fancy, this is going to just send the message key as a java.lang.String and the body of the message will also be of type java.lang.String, nothing fancy.

Our @PostMapping is going to simply be the endpoint with 2 query parameters:

firstname
lastname

So the URL will have the format:
http://localhost:8050/employee?firstname=Paula&lastname=Abdul

And when we call the API we will use Httpie with the following syntax:

http POST http://localhost:8070/employee firstname==Paula lastname==Abdul

Next we are just generating a random UUID as the message id and send this along with a concatenated string of the lastname and firstname to the employee topic:

String id = UUID.randomUUID().toString();
kafka.send("employee", id, String.format("%s, %s", lastname, firstname));

Before we start the application, lets tweak the configuration. In the src/main/resources/application.yml file, let's make it the same as:

spring:
  kafka:
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    bootstrap-servers:
      - localhost:9092

server:
  port: 8070

Now we can run the application and hit the endpoint:

./mvnw spring-boot:run
...
: Tomcat initialized with port(s): 8070 (http)
: Starting service [Tomcat]
: Starting Servlet engine: [Apache Tomcat/9.0.54]
: Initializing Spring embedded WebApplicationContext
: Root WebApplicationContext: initialization completed in 754 ms
: Tomcat started on port(s): 8070 (http) with context path ''
: Started DemoApplication in 1.522 seconds (JVM running for 1.841)

Call the endpoint:

$ http POST http://localhost:8070/employee firstname==Andrew lastname==Marshall
HTTP/1.1 201 
Connection: keep-alive
Content-Length: 0
Date: Wed, 27 Apr 2022 17:30:39 GMT
Keep-Alive: timeout=60
Location: /employee/830da346-38b9-4d5b-a051-a302c395333e

You can track the message in Kafka using the kafka-console-consumer.sh command:

$ $KAFKA_HOME/bin/kafka-console-consumer  --topic employee --bootstrap-server localhost:9092 --from-beginning
Marshall, Andrew

Next article we will setup a functional consumer using Spring Cloud Stream...

Open Source Data Ingestion Pipelines - Part 2

Anthony Ikeda — Tue, 06 Jul 2021 23:47:45 +0000

In the previous post we set up Apache Drill, Minio and PostgreSQL to set up a basic pipeline of data processing.

In this installment we are allso going to incorporate Kafka and show how we can merge the relational and realtime events.

Connecting to Kafka

With an existing kafka setup open up the Apache Drill Storage tab and click on Update next to the inactive Kafka source:

Sending Data to Kafka

Create a topic

Open a terminal and run the topic create command:

$ bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic purchase-events --partitions 1 --replication-factor 1
Created topic purchase-events.

Publish some JSON

Open a terminal in your KAFKA_HOME and run the kafka-consumer command:

$ bin/kafka-console-producer.sh --topic purchase-events --bootstrap-server localhost:9092
>

The scenario we will use is that once the main system has approved a payment, it will send a message to Kafka showing the transaction details and the approval status (SUCCESS or FAILED). There are some example JSON structures below:

{ "txn_id" :  12346, "store_id" :  "45873AD", "cust_id" :  "02145SYD", "amnt" :  122.49, "status" : "SUCCESS" }

{ "txn_id" :  12348, "store_id" :  "78534AS", "cust_id" :  "26354MLB", "amnt" :  16.01, "status" : "FAILED" }

{ "txn_id" :  12345, "store_id" :  "98369AS", "cust_id" :  "98764TGH", "amnt" :  34.99, "status" : "SUCCESS" }

{ "txn_id" :  12348, "store_id" :  "78534AS", "cust_id" :  "26354MLB", "amnt" :  16.01, "status" : "SUCCESS" }

{ "txn_id" :  12334, "store_id" :  "92374SS", "cust_id" :  "45268PER", "amnt" :  89.99, "status" : "FAILED" }

{ "txn_id" :  12352, "store_id" :  "98674DC", "cust_id" :  "25478MLB", "amnt" :  30.12, "status" : "SUCCESS" }

In your Drill Query console, run the following query:

select * from kafka.`purchase-events`;

You should get a table representing:

data
kafka topic
kafka partition id
message offset
message timestamp

Merging the RDBMS data with Realtime data

To actually make this data handy, what we will do is use the following scenario:

User has submitted their order and the order is stored in PostgreSQL
Our Order Service has submitted a request to the Payment Service to clear a payment for the order
The Payment Service sends the transaction details and payment status to Kafka once it has completed the request
We want to then view a quick report on the orders that are being placed and their status.
Successful orders will then be sent to stores to be fulfilled.

To satisfy this, we will take the order information from our Order service and process only successful orders.

(next article we will add actions for items that have failed)

Create the order PostgreSQL table

Run the following script to create the database, role and tables and insert some data:

create database sales_db;

create role sales_admin with password 'letmein' login;

grant all on database sales_db to sales_admin;

create table purchase_order (
  order_id bigint generated always as identity,
  order_total numeric(5,2),
  store_id varchar(12),
  customer_id varchar(12),
  order_status varchar(20) DEFAULT 'SUBMITTED'
);

alter table purchase_order add constraint pk_order PRIMARY KEY (order_id);

insert into purchase_order (order_total, store_id, customer_id) values (12.56, 'STORE_1', 'CUST_12');

insert into purchase_order (order_total, store_id, customer_id) values (122.65, 'STORE_15', 'CUST_89');

insert into purchase_order (order_total, store_id, customer_id) values (56.98, 'STORE_43', 'CUST_51');

insert into purchase_order (order_total, store_id, customer_id) values (35.00, 'STORE_1', 'CUST_61');

select * from purchase_order;
 order_id | order_total | store_id | customer_id | order_status 
----------+-------------+----------+-------------+--------------
        1 |       12.56 | STORE_1  | CUST_12     | SUBMITTED
        2 |      122.65 | STORE_15 | CUST_89     | SUBMITTED
        3 |       56.98 | STORE_43 | CUST_51     | SUBMITTED
        4 |       35.00 | STORE_1  | CUST_61     | SUBMITTED

We want some data that matches up to our database in the kafka topic so lets push the fresh data on the kafka-console-producer:

{ "txn_id" :  1, "store_id" :  "STORE_1", "cust_id" :  "CUST_12", "amnt" :  12.56, "status" : "SUCCESS" }

{ "txn_id" :  2, "store_id" :  "STORE_15", "cust_id" :  "CUST_89", "amnt" :  122.65, "status" : "FAILED" }

{ "txn_id" :  3, "store_id" :  "STORE_43", "cust_id" :  "CUST_51", "amnt" :  56.98, "status" : "FAILED" }

{ "txn_id" :  4, "store_id" :  "STORE_1", "cust_id" :  "CUST_61", "amnt" :  35.00, "status" : "SUCCESS" }

Create our query

Now that we have some data in our database and topic, let's create a topic to join the data:

select p.*, e.*
from kafka.`purchase-events` e
join salesdb.`purchase_order` p
on p.order_id = e.txn_id;

If successful, you should now have realtime data being matched with what is in your database!

Next article we will set up orchestration of all these events using another Apache open source project called Airflow.

Open Source Data Ingestion Pipeline

Anthony Ikeda — Tue, 15 Jun 2021 00:22:09 +0000

In this post I'm going to set up the foundation for a data ingestion pipeline integrating RDBMS and BLOB Storage.

Install and Set up Minio
Setup and configure PostgreSQL
Install and Set up Apache Drill
Your First Data Transfer

Install and Set up Minio

On macOS run the following command:

$ brew install minio/stable/minio

You will now have a new service configured for Minio:

$ brew services list
Name          Status  User         Plist
minio         stopped

You can start Minio with the following command:

$ brew services start minio
==> Successfully started `minio` (label: homebrew.mxcl.minio)
$ brew services list
Name          Status  User         Plist
minio         started anthonyikeda /Users/user_a/Library/LaunchAgents/homebrew.mxcl.minio.plist

You can access the web console on http://localhost:9000 in your web browser with the default credentials minioadmin/minioadmin

Part of the Minio tool kit is the client mc. In order to use the client from the command line, Minio expects an alias to be configured to the URL.

You can set up a new alias to connect to with:

$ mc alias minioserv http://localhost:9000 minioadmin minioadmin
Added `minioserv` successfully.

You should now be able to execute commands against your Minio instance:

$ mc mb minioserve/apachedrill
Bucket created successfully `minioserv/apachedrill`.
$ mc ls minioserv
[2021-06-11 16:15:29 PDT]     0B apachedrill/

In order to avoid using our admin user and privileges, let's create a new user that external apps will use.

From the command line, execute the following command to create a new user apachedrill with password letmeintotheparty:

$ mc admin user add minioserv apachedrill letmeintotheparty
Added user `apachedrill` successfully.

Our user, apachedrill can now be used to access Minio, however, they have no privileges to do anything. Thankfully, Minio is S3 compliant so we can use AWS policies to enable our user to interact with Minio.

Create a new file minio-privileges.json with the following content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
            "s3:ListBucket",
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
        ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::apachedrill/*",
        "arn:aws:s3:::apachedrill",
        "arn:aws:s3:::drilltmp/*",
        "arn:aws:s3:::drilltmp"
      ],
      "Sid": "BucketAccessForUser"
    }
  ]
}

Create the policy in Minio:

$ mc admin policy add minioserv drill-bucket-policy /path/to/minio-privileges.json

And allow our user to make use of the policy:

$ mc admin policy set minioserv drill-bucket-policy user=apachedrill
Policy `drill-bucket-policy` is set on user `apachedrill`

This allows your new user apachedrill to perform actions at a bucket level in Minio.

Set up and Configure PostgreSQL

If you haven't installed postgres before, we will follow a similar approach like we did with Minio:

$ brew install postgres
$ brew services start postgresql

You can access postgres with:

$ psql postgres
psql (13.3)
Type "help" for help.

postgres=#

For this article we will:

Create a database
Create a database user
Grant the user control over the database
Create some tables and data

Create the Database

Straight forward, we will create the database with:

postgres=# create database address_book_db;
CREATE DATABASE
postgres=#

Let's add a user and grant them control:

postgres=# create role drill_user with password 'letmein';
CREATE ROLE
postgres=# alter user drill_user with login;
postres=# grant all on database address_book_db to drill_user;
GRANT

For this article I have some tables and mock data here [https://github.com/anthonyikeda/contact-api/tree/main/src/main/resources/db/data]

Now for the heart of the article...

Install and Set up Apache Drill

Apache Drill integrates different datasources and uses a common syntax. Typically you'd run Drill in a clustered environment to handle the different streams of data processing; for this article we will stick with standalone.

Download the latest release from: https://drill.apache.org/download/

Move it to a local accessible folder, unzip it and open a command line to the drill home folder.

NOTE

You may need to install your JDBC Driver. You can easily download the JAR file from mvnrepository.org and copy the files to <DRILL_HOME>/jars/3rdparty/

Starting drill is easy:

$ bin/drill-embedded
Apache Drill 1.18.0
"Your Drill is the Drill that will pierce the heavens."
apache drill>

This starts the server and command line tool allowing you to browse the web console and execute commands from the terminal.

Open a web browser at: http://localhost:8047

Typically in a clustered environment you'd see more nodes, but since we are running in standalone mode you should see your single instance that is running.

Next we are going to connect to PostgreSQL.

Connect to PostgreSQL

Navigate to the Storage Tab and hit the Create button. This will bring up a rudimentary editor to paste our configuration.

Give the Storage option a name (address_book_db) and paste the following customizing for your environment:

{
  "type": "jdbc",
  "driver": "org.postgresql.Driver",
  "url": "jdbc:postgresql://localhost:5432/address_book_db",
  "username": "drill_user",
  "password": "letmein",
  "caseInsensitiveTableNames": false,
  "sourceParameters": {
    "minimumIdle": 5,
    "autoCommit": false,
    "connectionTestQuery": "select version() as postgresql_version",
    "dataSource.cachePrepStmts": true,
    "dataSource.prepStmtCacheSize": 250
  },
  "enabled": true
}

Hit Create and the new Storage should test the connection to the database and be ready to use.

If you return to the drill-cli you can now inspect your database connection:

apache drill> show databases;
+----------------------------------+
|           SCHEMA_NAME            |
+----------------------------------+
| addressbookdb.address_book_db    |
| addressbookdb.information_schema |
| addressbookdb.pg_catalog         |
| addressbookdb.public             |
| addressbookdb                    |
| sys                              |
+----------------------------------+

Now we will connect to the database and view the tables:

apache drill> use addressbookdb.address_book_db
+------+-----------------------------------------------------------+
|  ok  |                          summary                          |
+------+-----------------------------------------------------------+
| true | Default schema changed to [addressbookdb.address_book_db] |
+------+-----------------------------------------------------------+
1 row selected (0.13 seconds)
apache drill (addressbookdb.address_book_db)> show tables;
+-------------------------------+--------------------------------+
|      TABLE_SCHEMA             |            TABLE_NAME          |
+-------------------------------+--------------------------------+
| addressbookdb.address_book_db | flyway_schema_history_pk       |
| addressbookdb.address_book_db | flyway_schema_history_s_idx.   |
| addressbookdb.address_book_db | pk_address_id                  |
| addressbookdb.address_book_db | pk_contact_id                  |
| addressbookdb.address_book_db | address_address_id_seq         |
| addressbookdb.address_book_db | contacts_contact_id_seq        |
| addressbookdb.address_book_db | address                        |
| addressbookdb.address_book_db | contacts                       |
| addressbookdb.address_book_db | flyway_schema_history          |
+-------------------------------+--------------------------------+
9 rows selected (0.347 seconds)

apache drill (addressbookdb.address_book_db)>

We can query the tables using standard SQL syntax:

apache drill (addressbookdb.address_book_db)> select * from addressbookdb.contacts limit 10;
+------------+--------------------+-------------------+---------------------------+
| contact_id | contact_first_name | contact_last_name |   contact_email_address   |
+------------+--------------------+-------------------+---------------------------+
| 100        | Bert               | Gedling           | bgedling0@myspace.com     |
| 101        | Dacie              | Beddingham        | dbeddingham1@usda.gov     |
| 102        | Lina               | Wey               | lwey2@deviantart.com      |
| 103        | Savina             | Housbie           | shousbie3@google.com.au   |
| 104        | Delphinia          | Lente             | dlente4@delicious.com     |
| 105        | Zonda              | Blinder           | zblinder5@google.es       |
| 106        | Charmian           | Costley           | ccostley6@java.com        |
| 107        | Darcy              | Bulluck           | dbulluck7@flavors.me      |
| 108        | Tim                | Duesberry         | tduesberry8@google.com.br |
| 109        | Torr               | Cordero           | tcordero9@ftc.gov         |
+------------+--------------------+-------------------+---------------------------+
10 rows selected (0.228 seconds)

It's time to turn our attention back to Minio!

Wiring up Minio and pushing data

Open up the Storage tab in Drill again and create a new Datasource.

Give it a name (minio) and paste the following configuration:

{
  "type": "file",
  "connection": "s3a://apachedrill",
  "config": {
    "fs.s3a.secret.key": "letmeintotheparty",
    "fs.s3a.access.key": "apachedrill",
    "fs.s3a.connection.ssl.enabled": "false",
    "fs.s3a.endpoint": "http://127.0.0.1:9000",
    "fs.s3a.path.style.access": "false"
  },
  "workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "root": {
      "location": "/data",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "parquet": {
      "type": "parquet"
    },
    "avro": {
      "type": "avro",
      "extensions": [
        "avro"
      ]
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "pcap": {
      "type": "pcap",
      "extensions": [
        "pcap"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "pcapng": {
      "type": "pcapng",
      "extensions": [
        "pcapng"
      ]
    },
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "fieldDelimiter": "|"
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "fieldDelimiter": "\t"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ]
    },
    "syslog": {
      "type": "syslog",
      "extensions": [
        "syslog"
      ],
      "maxErrors": 10
    },
    "ltsv": {
      "type": "ltsv",
      "extensions": [
        "ltsv"
      ]
    },
    "hdf5": {
      "type": "hdf5",
      "extensions": [
        "h5"
      ],
      "defaultPath": null
    },
    "spss": {
      "type": "spss",
      "extensions": [
        "sav"
      ]
    },
    "shp": {
      "type": "shp",
      "extensions": [
        "shp"
      ]
    },
    "excel": {
      "type": "excel",
      "extensions": [
        "xlsx"
      ],
      "lastRow": 1048576
    }
  },
  "enabled": true
}

Key things to note:

S3 Connection

 "connection": "s3a://apachedrill",

This is the bucket we are targeting when we connect to Minio. Any workspaces we set up are subfolders of this bucket. Earlier we created a policy that outlined what can be done with this bucket and applied the policy to our user apachedrill.

S3 Configuration

"config": {
    "fs.s3a.secret.key": "letmeintotheparty",
    "fs.s3a.access.key": "apachedrill",
    "fs.s3a.connection.ssl.enabled": "false",
    "fs.s3a.endpoint": "http://127.0.0.1:9000",
    "fs.s3a.path.style.access": "false"
  }

Here we are configuring the connection to the Minio server, from which endpoint to use (fs.s3a.endpoint) to the credentials used to access Minio (fs.s3a.access.key, fs.s3a.secret.key)

It's important to override the fs.s3a.endpoint otherwise Drill will attempt to connect to S3 on AWS which we don't want right now.

Workspace Directories

"workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "root": {
      "location": "/data",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  }

These are the directories in the bucket we have nominated that we can put data in. When we push data to Minio from Drill, we will reference these directories as writeable targets.

Push data to Minio

What we will do next is create a materialized view from the data in postgresql and write the materialized view into Minio.

In the Apache Drill console type:

apache drill> use minio.root;
+------+----------------------------------------+
|  ok  |                summary                 |
+------+----------------------------------------+
| true | Default schema changed to [minio.root] |
+------+----------------------------------------+
1 row selected (0.195 seconds)
apache drill (minio.root)>

Now, we will create a table in Minio using data from PostgreSQL:

apache drill (minio.root)> create table contact_cities (contact_id, contact_first_name, contact_last_name, city) AS
2...............semicolon> select a.contact_id, c.contact_first_name, c.contact_last_name, a.city
3...............semicolon> from addressbookdb.address a
4...............semicolon> join addressbookdb.contacts c 
5...............semicolon> on c.contact_id = a.contact_id;
+----------+---------------------------+
| Fragment | Number of records written |
+----------+---------------------------+
| 0_0      | 1001                      |
+----------+---------------------------+
1 row selected (1.357 seconds)

What we have just done is taken the data in the form of our query (select a.contact_id, c.contact_first_name, c.contact_last_name, a.city from addressbookdb.address a join addressbookdb.contacts c on c.contact_id = a.contact_id;) and created a new view in Minio stored in parquet format.

If open your Minio browser and navigate to the apachedrill/data folder you should see a new folder: /contact_cities

In there should be a file called 0_0_0.parquet;

Let's see if we got the results we were after:

apache drill (minio.root)> select * from contact_cities limit 10;
+------------+--------------------+-------------------+------------+
| contact_id | contact_first_name | contact_last_name |    city    |
+------------+--------------------+-------------------+------------+
| 278        | Jania              | Kilalea           | Rochester  |
| 1          | Jonah              | Kent              | Lalor Park |
| 100        | Bert               | Gedling           | Newark     |
| 101        | Dacie              | Beddingham        | Sioux City |
| 102        | Lina               | Wey               | Lexington  |
| 103        | Savina             | Housbie           | Clearwater |
| 104        | Delphinia          | Lente             | Spokane    |
| 105        | Zonda              | Blinder           | New Castle |
| 106        | Charmian           | Costley           | Orlando    |
| 107        | Darcy              | Bulluck           | Dayton     |
+------------+--------------------+-------------------+------------+
10 rows selected (0.774 seconds)
apache drill (minio.root)> select count(*) from contact_cities;
+--------+
| EXPR$0 |
+--------+
| 1001   |
+--------+
1 row selected (0.491 seconds)

Looks like we were successful!

In the next article we will set up Kafka and query records coming from Kafka in Apache Drill!

Securing Angular and Quarkus with Keycloak Pt 3

Anthony Ikeda — Thu, 08 Oct 2020 18:01:42 +0000

Coming soon...# Securing Angular and Quarkus with Keycloak Pt 3

In our last article we left with some secure endpoints that were serving up content to the Angular application we built.

In this installment we will wrap up with using the roles and user profile to load user specfic data with our RewardsComponent.

Code for this article is here (API) and the UI code here.

Rewards Component

If you refer back to the previous article you'll see we created the RewardsComponent and endpoint but didn't do much with it. The goal of this article is to determine who is calling the endpoint through their user profile and load data using their User Id.

We will start with the API.

User Profile in the API

Quarkus makes idenitifying the user rather easy using the io.quarkus.security.identity.SecurityIdentity class.

We just need to inject it into our RewardsResource:

@Path("/v1/rewards")
public class RewardsResource {

    @Inject
    SecurityIdentity identity;

    @GET
    public Response getRewards() {
        return null;
    }
}

The SecurityIdentity gives you access to:

Permissions
Roles
Principal
Credentials

In order to determine who is making the secured request we can use:

String username = this.identity.getPrincipal().getName();

From an architecture standpoint the user id that Keycloak gives you is going to be constant. When it comes to your resource services will probably want to make sure either:

You use the identifier from the Keycloak server
You map the id from Keycloak to an internal identifier.

For the sake of this article we will just use the Keycloak identifier instead of creating a mapped id.

I've added in a database migration to our service to create accounts and store them in a table and added code to to retrieve them.

The data we are storing includes:

Reward Id	Owner	Reward Type
1	charlene.masters	'20% Discount'
2	charlene.masters	'BOGO Free'
3	mary.contrary	'10% Discount'
4	mary.contrary	'25% Discount'
5	mary.contrary	'invite a friend'

And the code that we will use server side includes:

RewardsResource.java

package com.cloudyengineering.pets;

import io.agroal.api.AgroalDataSource;
import io.quarkus.security.identity.SecurityIdentity;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.annotation.security.RolesAllowed;
import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Response;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.List;

@Path("/v1/rewards")
@Produces("application/json")
public class RewardsResource {

    private final Logger log = LoggerFactory.getLogger(RewardsResource.class);

    @Inject
    SecurityIdentity identity;

    @Inject
    AgroalDataSource defaultDataSource;

    public RewardsResource() {
    }

    @GET
    @RolesAllowed({"api-customer"})
    public Response getRewards() {
        String username = this.identity.getPrincipal().getName();
        List<Reward> rewards = new ArrayList<>();
        try {
            String sql = "select * from rewards where owner = ?";
            PreparedStatement stmt = defaultDataSource.getConnection().prepareStatement(sql);
            stmt.setString(1, username);
            ResultSet rs = stmt.executeQuery();
            while(rs.next()) {
                Reward reward = new Reward();
                reward.setRewardId(rs.getInt("reward_id"));
                reward.setOwner(rs.getString("owner"));
                reward.setRewardType(rs.getString("reward_type"));
                rewards.add(reward);
            }
            return Response.ok(rewards).build();
        } catch(Exception e) {
            e.printStackTrace();
            return Response.serverError().build();
        }
    }

}

What have we updated?

As you can see, aside from implementing the logic to load the data from a database, we've annotated our endpoint with the @RolesAllowed() annotation limiting access to users with the api-customer role. We've also used the SecurityIdentity to get the name of the user that is requesting the data allowing us to filter the query to that users' data only.

Let's update the User Interface!

rewards.component.html

<p>rewards works!</p>
<table>
    <tr>
        <th>Reward Id</th>
        <th>Reward Type</th>
    </tr>
    <tr *ngFor="let reward of rewards | async">
        <td>{{reward.reward_id}}</td>
        <td>{{reward.reward_type}}</td>
    </tr>
</table>

rewards.component.ts

import { Component, OnInit } from '@angular/core';
import { StoreService } from '../store.service';
import { map } from 'rxjs/operators';
import { Reward } from '../_model';
import { BehaviorSubject, Observable } from 'rxjs';

@Component({
  selector: 'app-rewards',
  templateUrl: './rewards.component.html',
  styleUrls: ['./rewards.component.css']
})
export class RewardsComponent implements OnInit {

  rewards = new BehaviorSubject<Reward[]>([]);

  constructor(private store: StoreService) { }

  ngOnInit(): void {
    const reward$ = this.store.getRewards().pipe( 
      map(results => {
        this.rewards.next(results);
      })
    );

    reward$.subscribe(data => data);
  }

}

Not much to report here except that we are using a BehaviourSubject to store the data from the StoreService to ensure async access to the data. This is why we are using the async pipe in the rewards.component.html.

Running our Code

If you run our service and UI you should now be able to log in as either Mary or Charlene and see only the rewards that belong to the user:

So that wraps up our journey using Keycloak, Quarkus and Angular.

Securing Angular and Quarkus with Keycloak Pt 2

Anthony Ikeda — Wed, 07 Oct 2020 17:54:46 +0000

In the previous article we created an initial user interface that enables Keycloak login and role recognition. The next step is to wire this up to a back end service to serve secure, role-based services.

Code for this article is located in Github here:

In a typical scenario I would implement 2 different types services:

Services for store employees to use serving up employee specific functions
Services for customer specific functions

I'd also implement multiple levels of integration. For now we will start with something simple, create a single Quarkus microservice and manage access through the roles we created.

So, let's kick it off!

Create the API Client

We are going back to the Keycloak admin UI to create a new client for the API to use:

Navigate to Clients, then click Create:

Once the client is created, navigate to the Roles tab to create the roles that we want the client to assume:

And we want to map these back to our users as in the previous article.

Navigate to Groups -> store-employees -> Role Mappings and add in the api-employee role:

Do the same for the api-customer role:

Next we want to start work on our service api.

Creating our Service

We will start from scratch and create a standard Quarkus service:

$ mvn io.quarkus:quarkus-maven-plugin:1.7.2.Final:create \
    -DprojectGroupId=com.brightfield.streams \
    -DprojectArtifactId=petstore-api

Some added dependencies that will make life easier include:

    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-resteasy-jackson</artifactId>
    </dependency>

Before we secure our application, let's get something running.

Implementing the Endpoints

There are 3 endpoints we will implement:

GET /pets
- accessible by both groups (store-employees and customers)
GET /sales
- accessible only by the store-employees
GET /rewards
- accessible by the logged in customer

In the project, let's create a couple of resources and representations:

PetResource.java

package com.cloudyengineering.pets;

import java.util.ArrayList;
import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Response;


@Path("/v1/pets")
@Produces("application/json")
public class PetResource {

    @GET
    public Response getPets() {
        List<Pet> pets = new ArrayList<>();

        return Response.ok(pets).build();
    }
}

SalesResource.java

package com.cloudyengineering.pets;

import java.util.ArrayList;
import java.util.List;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.core.Response;

@Path("/v1/admin")
@Produces("application/json")
public class SalesResource {

    @GET
    public Response getSales() {
        List<Transaction> transactions = new ArrayList<>();

        return Response.ok(transactions).build();
    }
}

Pet.java

package com.cloudyengineering.pets;

import com.fasterxml.jackson.annotation.JsonProperty;

public class Pet {

    @JsonProperty("pet_id")
    private Integer petId;

    @JsonProperty("pet_type")
    private String petType;

    @JsonProperty("pet_name")
    private String petName;

    @JsonProperty("pet_age")
    private Integer petAge;

    public Integer getPetId() {
        return petId;
    }

    public void setPetId(Integer petId) {
        this.petId = petId;
    }

    public String getPetType() {
        return petType;
    }

    public void setPetType(String petType) {
        this.petType = petType;
    }

    public String getPetName() {
        return petName;
    }

    public void setPetName(String petName) {
        this.petName = petName;
    }

    public Integer getPetAge() {
        return petAge;
    }

    public void setPetAge(Integer petAge) {
        this.petAge = petAge;
    }

}

Transaction.java

package com.cloudyengineering.pets;

import java.util.Date;

import com.fasterxml.jackson.annotation.JsonProperty;

public class Transaction {

    @JsonProperty("txn_id")
    private String transactionId;

    @JsonProperty("txn_amount")
    private Double transactionAmount;

    @JsonProperty("txn_date")
    private Date transactionDate;

    @JsonProperty("txn_method")
    private String transactionMethod;

    public String getTransactionId() {
        return transactionId;
    }

    public void setTransactionId(String transactionId) {
        this.transactionId = transactionId;
    }

    public Double getTransactionAmount() {
        return transactionAmount;
    }

    public void setTransactionAmount(Double transactionAmount) {
        this.transactionAmount = transactionAmount;
    }

    public Date getTransactionDate() {
        return transactionDate;
    }

    public void setTransactionDate(Date transactionDate) {
        this.transactionDate = transactionDate;
    }

    public String getTransactionMethod() {
        return transactionMethod;
    }

    public void setTransactionMethod(String transactionMethod) {
        this.transactionMethod = transactionMethod;
    }
}

As you can see, these are just placeholders so let's fill in some logic:

    @GET
    public Response getPets() {
        List<Pet> pets = new ArrayList<>();
        Pet pet1 = new Pet();
        pet1.setPetId(1);
        pet1.setPetAge(6);
        pet1.setPetName("Oliver");
        pet1.setPetType("Dog");

        Pet pet2 = new Pet();
        pet2.setPetId(2);
        pet2.setPetAge(1);
        pet2.setPetName("Buster");
        pet2.setPetType("Cat");

        Pet pet3 = new Pet();
        pet3.setPetId(3);
        pet3.setPetAge(2);
        pet3.setPetName("Violet");
        pet3.setPetType("Bird");

        pets = Lists.asList(pet1, new Pet[]{pet2, pet3});

        return Response.ok(pets).build();
    }

TransactionResource.java

    @GET
    public Response getSales() {
        List<Transaction> transactions = new ArrayList<>();

        Transaction txn1 = new Transaction();
        txn1.setTransactionId(UUID.randomUUID().toString());
        txn1.setTransactionAmount(12.56);
        txn1.setTransactionDate(Date.from(Instant.now()));
        txn1.setTransactionMethod("Cash");

        Transaction txn2 = new Transaction();
        txn2.setTransactionId(UUID.randomUUID().toString());
        txn2.setTransactionAmount(56.16);
        txn2.setTransactionDate(Date.from(Instant.now()));
        txn2.setTransactionMethod("Credit Card");

        Transaction txn3 = new Transaction();
        txn3.setTransactionId(UUID.randomUUID().toString());
        txn3.setTransactionAmount(88.99);
        txn3.setTransactionDate(Date.from(Instant.now()));
        txn3.setTransactionMethod("Credit Card");

        transactions = Lists.asList(txn1, new Transaction[]{txn2, txn3});

        return Response.ok(transactions).build();
    }

Let's quickly give it a test!

$ ./mvnw quarkus:dev
Listening for transport dt_socket at address: 5005
__  ____  __  _____   ___  __ ____  ______
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2020-10-04 18:30:09,620 INFO  [io.quarkus] (Quarkus Main Thread) pet-store-api 1.0-SNAPSHOT on JVM (powered by Quarkus 1.8.1.Final) started in 3.296s. Listening on: http://0.0.0.0:
8080
2020-10-04 18:30:09,622 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2020-10-04 18:30:09,622 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, resteasy, resteasy-jackson]

$ http :8080/v1/pets
HTTP/1.1 200 OK
Content-Length: 188
Content-Type: application/json

[
    {
        "pet_age": 6,
        "pet_id": 1,
        "pet_name": "Oliver",
        "pet_type": "Dog"
    },
    {
        "pet_age": 1,
        "pet_id": 2,
        "pet_name": "Buster",
        "pet_type": "Cat"
    },
    {
        "pet_age": 2,
        "pet_id": 3,
        "pet_name": "Violet",
        "pet_type": "Bird"
    }
]

$ http :8080/v1/admin
HTTP/1.1 200 OK
Content-Length: 357
Content-Type: application/json

[
    {
        "txn_amount": 12.56,
        "txn_date": 1601862235453,
        "txn_id": "cb09b51d-541a-45b5-9c27-13a09b480dfd",
        "txn_method": "Cash"
    },
    {
        "txn_amount": 56.16,
        "txn_date": 1601862235454,
        "txn_id": "2895fe10-31ae-417a-8d7d-28ccdf0fa08b",
        "txn_method": "Credit Card"
    },
    {
        "txn_amount": 88.99,
        "txn_date": 1601862235454,
        "txn_id": "afbec85c-b919-40e4-9b02-9e5779534b0b",
        "txn_method": "Credit Card"
    }
]

Hmmm... those dates don't look too friendly, let's change them:

Transaction.java

    @JsonProperty("txn_date")
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "dd-MM-yyyy hh:mm:ss")
    private Date transactionDate;

And again...

$ http :8080/v1/admin
HTTP/1.1 200 OK
Content-Length: 381
Content-Type: application/json

[
    {
        "txn_amount": 12.56,
        "txn_date": "05-10-2020 01:46:17",
        "txn_id": "8d5e875b-06ec-4cf6-b357-4e14525b831c",
        "txn_method": "Cash"
    },
    {
        "txn_amount": 56.16,
        "txn_date": "05-10-2020 01:46:17",
        "txn_id": "5764bef6-1c78-4efd-8016-119759b263c5",
        "txn_method": "Credit Card"
    },
    {
        "txn_amount": 88.99,
        "txn_date": "05-10-2020 01:46:17",
        "txn_id": "67a5f766-4a9f-4fa4-8e22-59ecf9171e55",
        "txn_method": "Credit Card"
    }
]

Nice! It works but is isn't quite secure yet.

Securing the API

There will be 2 dependencies we will be adding to secure our service:

pom.xml

  <dependencies>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-oidc</artifactId>
    </dependency>

    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-keycloak-authorization</artifactId>
    </dependency>
  </dependencies>

application.properties

quarkus.oidc.auth-server-url=http://localhost:8081/auth/realms/petshop-realm
quarkus.oidc.client-id=pet-store-api
quarkus.oidc.credentials.secret=itsasecret
quarkus.oidc.authentication.scopes=profile

quarkus.http.cors.origins=http://localhost:4200
quarkus.http.cors.methods=GET,OPTIONS
quarkus.http.cors=true

As you can see we have declared the client id petshop-api but we've also supplied the secret itsasecret, let's make sure that is configured!

Navigate to: Clients -> pet-store-api

Change Access Type to bearer only and hit save. You should see the tabs above change!

Next navigate to Credentials in the tabs above and you should see your client secret:

Let's copy this generated secret to our Pet Store API:

quarkus.oidc.auth-server-url=http://localhost:8081/auth/realms/petshop-realm
quarkus.oidc.client-id=pet-store-api
quarkus.oidc.credentials.secret=05da844f-c975-4571-8767-cbc8078e7b64
quarkus.oidc.authentication.scopes=profile

If we run our API now and try and access it we should get a 401 Unauthorized:

$ http :8080/v1/admin
HTTP/1.1 200 OK
Content-Length: 381
Content-Type: application/json

[
    {
        "txn_amount": 12.56,
        "txn_date": "06-10-2020 09:34:58",
        "txn_id": "a19f4089-5f20-4355-ab4a-a18367347d6d",
        "txn_method": "Cash"
    },
    {
        "txn_amount": 56.16,
        "txn_date": "06-10-2020 09:34:58",
        "txn_id": "6457746c-9c07-4cea-b8c5-fb30dc4e2f3d",
        "txn_method": "Credit Card"
    },
    {
        "txn_amount": 88.99,
        "txn_date": "06-10-2020 09:34:58",
        "txn_id": "09125408-0787-41d0-9976-8e56c5937bb3",
        "txn_method": "Credit Card"
    }
]

Wait a minute! That's not right! Oh I remember, we haven't specified security around our endpoints! Let's update the code:

SalesResource.java

@Path("/v1/admin")
@Produces("application/json")
public class SalesResource {

    @GET
    @RolesAllowed({"api-employee"})
    public Response getSales() {
        List<Transaction> transactions;

        Transaction txn1 = new Transaction();
        txn1.setTransactionId(UUID.randomUUID().toString());
        txn1.setTransactionAmount(12.56);
        txn1.setTransactionDate(Date.from(Instant.now()));
        txn1.setTransactionMethod("Cash");

        Transaction txn2 = new Transaction();
        txn2.setTransactionId(UUID.randomUUID().toString());
        txn2.setTransactionAmount(56.16);
        txn2.setTransactionDate(Date.from(Instant.now()));
        txn2.setTransactionMethod("Credit Card");

        Transaction txn3 = new Transaction();
        txn3.setTransactionId(UUID.randomUUID().toString());
        txn3.setTransactionAmount(88.99);
        txn3.setTransactionDate(Date.from(Instant.now()));
        txn3.setTransactionMethod("Credit Card");

        transactions = Lists.asList(txn1, new Transaction[]{txn2, txn3});

        return Response.ok(transactions).build();
    }
}

PetResource.java

@Path("/v1/pets")
@Produces("application/json")
public class PetResource {

    @GET
    @RolesAllowed({"api-customer"})
    public Response getPets() {
        List<Pet> pets;
        Pet pet1 = new Pet();
        pet1.setPetId(1);
        pet1.setPetAge(6);
        pet1.setPetName("Oliver");
        pet1.setPetType("Dog");

        Pet pet2 = new Pet();
        pet2.setPetId(2);
        pet2.setPetAge(1);
        pet2.setPetName("Buster");
        pet2.setPetType("Cat");

        Pet pet3 = new Pet();
        pet3.setPetId(3);
        pet3.setPetAge(2);
        pet3.setPetName("Violet");
        pet3.setPetType("Bird");

        pets = Lists.asList(pet1, new Pet[]{pet2, pet3});

        return Response.ok(pets).build();
    }
}

And try again...

$ http :8080/v1/admin
HTTP/1.1 401 Unauthorized
Content-Length: 0
$ http :8080/v1/pets
HTTP/1.1 401 Unauthorized
Content-Length: 0

Great! It's secure!

Ready to integrate the User Interface?

Connecting our UI with the secure services

When we last visited the User Interface, we were able to get user login working as well as ensure that different roles had different views. The changes we are going to make to the UI include:

Add a new view to list Pets for customers and employees
Add a new view to list Sales for employees
Add a new view to list Rewards for customers
Add in an AuthGuard to ensure different roles don't try and cheat and bypass the URIs

Creating the Views

Let's start creating new views!

Start by creating the new PetsComponent, SalesComoponent and RewardsComoponent:

$ ng g c pet
CREATE src/app/pet/pet.component.css (0 bytes)
CREATE src/app/pet/pet.component.html (18 bytes)
CREATE src/app/pet/pet.component.spec.ts (605 bytes)
CREATE src/app/pet/pet.component.ts (263 bytes)
UPDATE src/app/app.module.ts (947 bytes)
$ ng g c sales
CREATE src/app/sales/sales.component.css (0 bytes)
CREATE src/app/sales/sales.component.html (20 bytes)
CREATE src/app/sales/sales.component.spec.ts (619 bytes)
CREATE src/app/sales/sales.component.ts (271 bytes)
UPDATE src/app/app.module.ts (1025 bytes)
$ ng g c rewards
CREATE src/app/rewards/rewards.component.css (0 bytes)
CREATE src/app/rewards/rewards.component.html (22 bytes)
CREATE src/app/rewards/rewards.component.spec.ts (633 bytes)
CREATE src/app/rewards/rewards.component.ts (279 bytes)
UPDATE src/app/app.module.ts (1111 bytes)
$ ng g guard auth
? Which interfaces would you like to implement? CanActivate
CREATE src/app/auth.guard.spec.ts (331 bytes)
CREATE src/app/auth.guard.ts (457 bytes)

I'm not going to go full on MVC approach here so we will create 1 service to call the different APIs:

$ ng g s store
CREATE src/app/store.service.spec.ts (352 bytes)
CREATE src/app/store.service.ts (134 bytes)

Let's focus on the 2 main areas:

StoreService
AuthGuard

StoreService

Based on our original endpoints, lets create 3 functions:

getPets(): Observable<Pet[]>
getSales(): Observable<Transaction[]>
getRewards(): Observable<Reward[]>

store.service.ts

import { Injectable } from '@angular/core';
import { Observable } from 'rxjs';
import { Pet, Reward, Transaction } from './_model';
import { environment as env } from '../environments/environment';
import { HttpClient } from '@angular/common/http';

@Injectable({
  providedIn: 'root'
})
export class StoreService {

  constructor(private http : HttpClient) { }

  getPets(): Observable<Pet[]> {
    const uri = `${env.api_host}/v1/pet`;
    return this.http.get<Pet[]>(uri);
  }

  getSales(): Observable<Transaction[]> {
    const uri = `${env.api_host}/v1/sales`;
    return this.http.get<Transaction[]>(uri);
  }

  getRewards(): Observable<Reward[]> {
    const uri = `${env.api_host}/v1/rewards`;
    return this.http.get<Reward[]>(uri);
  }
}

Let's wire in the service to our components and render the results:

sales.component.ts

import { Component, OnInit } from '@angular/core';
import { StoreService } from '../store.service';
import { map, catchError } from 'rxjs/operators';
import { Transaction } from '../_model';
import { of } from 'rxjs';

@Component({
  selector: 'app-sales',
  templateUrl: './sales.component.html',
  styleUrls: ['./sales.component.css']
})
export class SalesComponent implements OnInit {

  sales: Transaction[];

  constructor(private store: StoreService) { }

  ngOnInit(): void {
    const sale$ = this.store.getSales().pipe(
      map(results => {
        this.sales = results;
      }),
      catchError(error => {
        console.log(error);
        return of([]);
      })
    );

    sale$.subscribe(data => data);
  }

}

sale.component.html

<p>sales works!</p>
<table>
    <tr>
        <th>Transaction ID</th>
        <th>Transaction Date</th>
        <th>Transaction Amount</th>
        <th>Payment Method</th>
    </tr>
    <tr *ngFor="let sale of sales">
        <td>{{sale.txn_id}}</td>
        <td>{{sale.txn_date}}</td>
        <td>{{sale.txn_amount}}</td>
        <td>{{sale.txn_method}}</td>
    </tr>
    <tr *ngIf="sales === undefined">
        <td colspan="4">No data</td>
    </tr>
</table>

pet.component.ts

import { Component, OnInit } from '@angular/core';
import { StoreService } from '../store.service';
import { map } from 'rxjs/operators';
import { Reward } from '../_model';

@Component({
  selector: 'app-rewards',
  templateUrl: './rewards.component.html',
  styleUrls: ['./rewards.component.css']
})
export class RewardsComponent implements OnInit {

  rewards: Reward[];

  constructor(private store: StoreService) { }

  ngOnInit(): void {
    this.store.getRewards().pipe( 
      map(results => {
        this.rewards = results;
      })
    );
  }

}

pet.component.html

<p>pet works!</p>
<table>
    <tr>
        <th>Pet ID</th>
        <th>Pet Type</th>
        <th>Pet Age</th>
        <th>Pet Name</th>
    </tr>
    <tr *ngFor="let pet of pets">
        <td>{{pet.pet_id}}</td>
        <td>{{pet.pet_type}}</td>
        <td>{{pet.pet_age}}</td>
        <td>{{pet.pet_name}}</td>
    </tr>
    <tr *ngIf="pets === undefined">
        <td colspan="4">No data</td>
    </tr>
</table>

As you can see we are storing the hostname for the api in the environment object and constructing our URIs from it.

You can see the source code for the structure of the model classes.

Let's log in as Charlene Masters and see what resolves:

As you can see we get a result based upon the allocated roles. But how dos the API know what roles the user has? If you open the developer console and select the Network tab, take a look at the XHR Request to the server:

If you take a look, you can see the bearer token being passed as part of the API request!

Because we've mapped the pet-store-api client roles to the Group, these roles are passed across as part of the bearer token to the API, honoring the access the user has.

Setting up our Guard

So in any standard application, Charlene should not be able to access the Sales view at all. In the next section we are going to setup the AuthGuard to check her roles and ensure we decline access to the SalesComponent.

First in order to use Keycloak to guard our route, we need to make some changes to our AuthGuard:

auth.guard.ts

import { Injectable } from '@angular/core';
import { ActivatedRouteSnapshot, RouterStateSnapshot, Router } from '@angular/router';
import { KeycloakAuthGuard, KeycloakService } from 'keycloak-angular';

@Injectable({
  providedIn: 'root'
})
export class AuthGuard extends KeycloakAuthGuard {

  constructor(protected router: Router, protected service: KeycloakService) {
    super(router, service)
  }

  isAccessAllowed(route: ActivatedRouteSnapshot,state: RouterStateSnapshot) : Promise<boolean> {
    return new Promise((resolve, reject) => {
        resolve(true)
    });
  }  
}

Here we are using the Keycloak wrapper to ensure the correct authenticated calls back to the Keycloak server are made.

What's important in this class is the isAccessAllowed() method. This is the entrypoint that will dictate if the user can activate the component.

Let's do a check on the path and see if they should access the sales data:

auth.guard.ts

  isAccessAllowed(route: ActivatedRouteSnapshot,state: RouterStateSnapshot) : Promise<boolean> {
    return new Promise((resolve, reject) => {
      const userRoles: string[] = this.service.getUserRoles();
      console.log(`Roles: ${userRoles}`);

      if (state.url === '/sales' && userRoles.indexOf('employee') >= 0) {
        console.log('Permission allowed');
        resolve(true)
      } else {
        console.log('Permission not allowed');
        resolve(false);
      }

    });

Setting up this guard in our app-routing.module.ts is as simple as:

const routes: Routes = [
  { path: 'pets', component: PetComponent },
  { path: 'rewards', component: RewardsComponent },
  { path: 'sales', component: SalesComponent, canActivate: [AuthGuard] }
];

Now if you access the sales.component as Charlene you should be able to see a message Permission not allowed in the console and the component is never activated.

NOTE

You can see we are printing out the roles the user has and if you take a look you'll notice that
Charlene has the following roles: api-customer,customer,manage-account,manage-account-links,view-profile,offline_access,uma_authorization

So we've now been able to secure our User Interface and our backend services. There is, of course, more you can do with this but it should give you a head start on secure applcations.

In Part 3, we will wire up the RewardsComponent and load user specific information!

Securing Angular and Quarkus with Keycloak Pt 1

Anthony Ikeda — Wed, 23 Sep 2020 04:34:16 +0000

Apologies

First I want to apologise to my readers, it seems a whole chunk of my article went missing while trying to post the sample code and escape the curly braces that ended up clashing with the liquid tags.

I've restore the missing content below.

Let's get into the article!

This will be part 1 of a series of posts where we will secure a stack from an Angular 10 UI to a backend resource server with Keycloak.

In this article, we will cover the initial user interface creating a realm, groups, users, permissions and a client for the UI to use as authentication and authorization.

The code for this first article is here: https://github.com/cloudy-engineering/pet-store-ui

First we need to set up our environment which will require the following:

docker-compose
Angular 10
Angular cli

Auth Model

The overall Auth model we will be setting up for this series consists of 2 main components:

A client for Single sign on for our User Interface
A client for the Resource Server and mapping it to our single sign on model

Setting up Keycloak

We will be using the dockerized version and we will also ensure we have a persistent state using PostgreSQL as our database.

version: "3.8"
services:
  keycloak:
    image: jboss/keycloak:latest
    environment:
      KEYCLOAK_USER: admin
      KEYCLOAK_PASSWORD: superSecret
      DB_VENDOR: postgres
      DB_ADDR: keycloak-db
      DB_DATABASE: keycloak
      DB_USER: keycloak
      DB_PASSWORD: keycloak
    depends_on:
      - keycloak-db
    ports:
      - 8081:8080

  keycloak-db:
    image: postgres:alpine
    environment:
      POSTGRES_PASSWORD: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_DB: keycloak
    volumes:
      - ./postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres']
      interval: 10s
      timeout: 5s
      retries: 5

To start up Keycloak, navigate to this file and run:

$ docker-compose up
...
INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 10.0.2 (WildFly Core 11.1.1.Final) started in 21588ms - Started 690 of 995 services (708 services are lazy, passive or on-demand)

Once Keycloak and PostgreSQL has started you can access the first user interface at http://localhost:8081:

From here we want to access the Administration Console:

The username and password were defined in the docker-compose.yml file as:

Username: admin
Password: superSecret

This will take you to our initial master realm:

Create a new Realm

Since we are starting from scratch with this article, we will create a brand new realm to work with. On the 'Master' drop down select "Add Realm":

We will call this new realm petshop-realm and click Create.

Petshop Realm

The petshop realm will be how we manage our applications and users. Here we will configure:

Users
Groups
Permissions
Client identifiers

These will connect up our User Interface, backend service and the users we manage in our realm.

First let's create some Users. Select 'Users' under the left navigation Manage section and create a couple of users:

For each user, select Edit → Credentials then add a password (e.g. letmein) and deactivate Temporary Password. Click Set Password.

Next we will create 2 groups:

customers
store-employees

Once the groups have been created, let's add some user. For the store-employees group, add Bob Small. To our customers group, let's add Charlene and Mary.

Before we do any more in Keycloak, let's create a quick store app with Angular 10.

Pet Store UI

In this article we will start with the user interface and enable the users we created to login and have the application enable certain functions based on the group the user belongs to.

Let's get our Angular app primed:

$ ng new pet-store --routing --style=css
CREATE pet-store/README.md (1026 bytes)
CREATE pet-store/.editorconfig (274 bytes)
CREATE pet-store/.gitignore (631 bytes)
...
CREATE pet-store/e2e/src/app.po.ts (301 bytes)
✔ Packages installed successfully.
Successfully initialized git.

Next we are going to create the initial home page. The home page will perform the following tasks:

If the user is a customer, show a left navigation to browse the store
If the user is a store-employee, the left navigation will show links to inventory

But first, open the project in the IDE of your choice, open the app.component.html page and remove everything. For now we won't be using routing.

Next create 2 components:

$ ng g c store-nav
CREATE src/app/store-nav/store-nav.component.css (0 bytes)
CREATE src/app/store-nav/store-nav.component.html (24 bytes)
CREATE src/app/store-nav/store-nav.component.spec.ts (641 bytes)
CREATE src/app/store-nav/store-nav.component.ts (286 bytes)
UPDATE src/app/app.module.ts (485 bytes)
$ ng g c admin-nav
CREATE src/app/admin-nav/admin-nav.component.css (0 bytes)
CREATE src/app/admin-nav/admin-nav.component.html (24 bytes)
CREATE src/app/admin-nav/admin-nav.component.spec.ts (641 bytes)
CREATE src/app/admin-nav/admin-nav.component.ts (286 bytes)
UPDATE src/app/app.module.ts (577 bytes)
$

We will create a very quick customization of both as below:

store-nav.component.html

<p>Store Navigation</p>
<ul>
  <li>Latest Deals</li>
  <li>Puppies</li>
  <li>Kittens</li>
  <li>Exotic pets</li>
</ul>

app-nav.component.html

<p>Store Details</p>
<ul>
  <li>Inventory</li>
  <li>Sales</li>
  <li>Reporting</li>
</ul>

Next, let's add these components to our app.component.html page:

app.component.html

<app-store-nav></app-store-nav>
<app-admin-nav></app-admin-nav>

If we run our application, we should see both items being displayed:

Adding Login Capability

For now that's as far as we can go with the User Interface. We next need to set up Keycloak to enable people to login to the application.

Access the Keycloak admin console and go to clients:

Click on Create

For our new client, we will name it petstore-portal. For the Root URL we will use http://localhost:4200 for now. Click Save.

Give the new client a name and set the Login Theme to Keycloak:

Next, in the top navigation, click Roles:

Let's create 2 new Roles:

customer
store-employee

Now, we want to map our roles to the groups we created earlier. Click on Groups in the left navigation:

And first select the customers group and click Edit.

Click on Role Mappings then in the Client Roles text input, type/select petstore-portal

In the Available Roles click Customer and then click Add Selected.

Do the same for the Store Employees group, but select store-employee as the role to add.

If you click on Members, you should see the users we assigned to the Groups in the beginning:

Great! Now it's time to wire up our user interface to let the people in!

Enabling Single Sign On

For connectivity to Keycloak, we are going to use the keycloak-angular and keycloak-js libraries:

$ yarn add keycloak-angular keycloak-js@10.0.2
yarn add v1.22.5
[1/4] 🔍 Resolving packages...
[2/4] 🚚 Fetching packages...
[3/4] 🔗 Linking dependencies...
[4/4] 🔨 Building fresh packages...
success Saved lockfile.
success Saved 3 new dependencies.
info Direct dependencies
├─ keycloak-angular@8.0.1
└─ keycloak-js@10.0.2
info All dependencies
├─ js-sha256@0.9.0
├─ keycloak-angular@8.0.1
└─ keycloak-js@11.0.2
✨ Done in 4.99s.

NOTE

When adding keycloak-js make sure you match up the correct version of the library with the version of Keycloak you are running.

If you find 404's popping up when trying to login in to your Angular application then this is probably the reason.

You can check the version of your Keycloak server by accessing: Admin → Server Info

It will be the first entry as Server Version.

Next let's set up some configuration parameters. Open the src/environment/environment.ts file and add the following configuration:

export const environment = {
  production: false,
  keycloak: {
    issuer: 'http://localhost:8081/auth/',
    realm: 'petshop-realm',
    clientId: 'petstore-portal'
  }
};

Next we want to load this configuration so we create an initializer:

$ ng g s initializer
CREATE src/app/initializer.service.spec.ts (382 bytes)
CREATE src/app/initializer.service.ts (140 bytes)
$

Now let's implement the initializer:

initializer.service.ts

import { KeycloakService } from 'keycloak-angular';
import { environment as env} from '../environments/environment';

export function initializer(keycloak: KeycloakService): () => Promise<any> {
  return (): Promise<any> => {
    return new Promise(async (resolve, reject) => {
      try {
        await keycloak.init({
          config: {
            url: env.keycloak.issuer,
            realm: env.keycloak.realm,
            clientId: env.keycloak.clientId,
          },
          loadUserProfileAtStartUp: true,
          initOptions: {
            onLoad: 'login-required'
          },
          bearerExcludedUrls: []
        });
        resolve();
      } catch(error) {
        reject(error);
      }
    });
  };
}

So what are we doing here? On Line 8, we are initializing the KeyCloak service with the settings we placed in our environment.ts file. This sets the auth server we plan to use (localhost:8081), the realm (petshop-realm) and the client (petstore-portal). We are also instructing Keycloak to load the User Profile at startup and ensure the user logs in initially.

One last thing we need to do is bootstrap our initializer in the app.module.ts file:

app.module.ts

import { BrowserModule } from '@angular/platform-browser';
import { APP_INITIALIZER, NgModule } from '@angular/core';

import { AppRoutingModule } from './app-routing.module';
import { AppComponent } from './app.component';
import { StoreNavComponent } from './store-nav/store-nav.component';
import { AdminNavComponent } from './admin-nav/admin-nav.component';
import { initializer } from './initializer.service';
import { KeycloakAngularModule, KeycloakService } from 'keycloak-angular';

@NgModule({
  declarations: [
    AppComponent,
    StoreNavComponent,
    AdminNavComponent
  ],
  imports: [
    BrowserModule,
    AppRoutingModule,
    KeycloakAngularModule,
  ],
  providers: [
    {
      provide: APP_INITIALIZER,
      useFactory: initializer,
      deps: [KeycloakService],
      multi: true
    }
  ],
  bootstrap: [AppComponent]
})
export class AppModule { }

Here we are importing the KeycloakAngularModule and KeycloakService and making sure we use both.

Now if we build and run our Angular application and access it at http://localhost:4200/ you should find yourself redirected to the Keycloak login page:

You can use any of the login credentials you created earlier to then authenticate and be redirected to our app.component.html page:

Congrats, you are now securely logged in to your application!

Let's see if we can limit the view to the specified role.

Role Based Views

When we set up Bob earlier, we added him to our store-employees group. In our Keycloak initializer we indicated we want to load the User Profile when they logged in. Using the Keycloak Service, we can get the roles the user currently belongs to a limit what they can access:

var roles: string[] = this.keycloakService.getUserRoles();

Let's update the app.component.ts to retrieve the roles and make them accessible to our page:

app.component.ts

import { Component, OnInit } from '@angular/core';
import { KeycloakService } from 'keycloak-angular';

@Component({
selector: 'app-root',
templateUrl: './app.component.html',
styleUrls: ['./app.component.css']
})
export class AppComponent implements OnInit {
  title = 'pet-store';
  roles: string[];
  username: string;

  constructor(private keycloak: KeycloakService) {}

  ngOnInit() {
    this.roles = this.keycloak.getUserRoles();
    this.keycloak.loadUserProfile().then(profile => {
      this.username = `${profile.firstName} ${profile.lastName}`;
    });
  }

}

Now, let's put some conditions in the UI to only allow the respective roles to access the different lists:

app.component.html

    <div *ngIf="username">Welcome username </div>
    <div *ngIf="roles.includes('customer')">
      <app-store-nav></app-store-nav>
    </div>
    <div *ngIf="roles.includes('store-employee')">
      <app-admin-nav></app-admin-nav>
    </div>

As you can see in the app.component.ts, we've injected the KeycloakService and use it to get the list of roles the user has (you may see more than the one we allocated). In our user interface we wrap our customer navigation components with a div tag and put conditions in place to limit the visibility to the specified roles. If you access the application in a web browser now you should see only what the user is authorized to access:

TIP

If you are seeing errors in the JavaScript console about Refused to frame 'http://localhost:8081/` then you can correct this by updating the Content-Security-Policy in Keycloak.

Navigate to Realm Settings → Security Settings and update the Content-Security-Policy with:

frame-src 'self' http://localhost:4200; frame-ancestors 'self' http://localhost:4200; object-src none;

This will ensure that, much like CORS, that localhost:4200 can be identified as being able to load content from the Keycloak Server.

That's it for now, in the next article we will wire up the Quarkus Microservice and communicate with it securely using OIDC.

Quarkus, WebSockets and Kafka

Anthony Ikeda — Mon, 14 Sep 2020 22:32:01 +0000

So previously, we looked at how we can quickly get Quarkus up and running and create a basic UI to send messages back and forth from the client to the server. In this article we will take it to the next level and introduce Kafka as a messaging platform and have incoming messages from a topic pushed directly to the User Interface.

There is no true session management in this article, we can cover that in the future, but this does demonstrate how easy it is to manage some basic users and broadcast to them all.

Getting Kafka Up and Running

For this to work, we are going to need a Kafka instance up and running, so we will start with that.

This are the requirements for this article:

Java 11
Apache ZooKeeper
Kafka 2.3.0
The source code will be on this branch

We will refer to the location you unzipped Kafka as KAFKA_HOME

Starting ZooKeeper

Once you've downloaded zookeeper, unzip it to a directory and ensure Java 11 is the current JDK.

Next, we want to create a conf/zoo.cfg file with the following properties:

cfg/zoo.cfg

tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=60

You can configure the dataDir to any location as long as the server can write to that directory. You can then start ZooKeeper with:

$ bin/zkServer.sh start conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo.cfg
Starting zookeeper ... STARTED

Next we will setup Kafka.

Kafka Up and Running

To get Kafka running we first need to make sure we have Java 11 set as the JDK.

Next start up Kafka with:

$ bin/kafka-server.sh start config/server.properties
INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
INFO starting (kafka.server.KafkaServer) [2020-09-08 19:04:53,486] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
INFO Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT (org.apache.zookeeper.ZooKeeper)
INFO Client environment:host.name=localhost (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.version=14.0.2 (org.apache.zookeeper.ZooKeeper)
INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
...
INFO Log directory /tmp/kafka-logs not found, creating it. (kafka.log.LogManager)
INFO Loading logs. (kafka.log.LogManager)
INFO Logs loading complete in 10 ms. (kafka.log.LogManager)
INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
INFO [SocketServer brokerId=0] Created data-plane acceptor and processors for endpoint : EndPoint(null,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)

There will be a bunch of mesages, but the more important one is the listener that was started: EndPoint(null,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)

This indicates we can connect to Kafka using a non-secured connection on port 9092

Create our Topic

We need to manually create a topic that we can read and write from. Open up a terminal, navigate to the KAFKA_HOME directory and execute the following command:

$ bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic chat-messages --partitions 1 --replication-factor 1
Created topic chat-messages.

This will create a new topic for us called chat-messages.

Updating the WebSocket API

In order to continue, we will need some more dependencies in our WebSocket API to connect to Kafka.

io.quarkus:quarkus-kafka-streams
org.testcontainers:testcontainers
org.testcontainers:kafka

Update the pom.xml dependencies with:

pom.xml

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-kafka-streams</artifactId>
</dependency>

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.14.3</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>kafka</artifactId>
    <version>1.14.3</version>
    <scope>test</scope>
</dependency>

Configure the Kafka Connection

Next we want to make sure we have the application configured to connect to our Kafka server. Open the src/main/resources/application.properties and make the following changes:

quarkus.kafka-streams.application-server=localhost:8011
quarkus.kafka-streams.application-id=${quarkus.application.name}
quarkus.kafka-streams.bootstrap-servers=${KAFKA_HOST:localhost}:${KAFKA_PORT:9092}
quarkus.kafka-streams.topics=chat-messages

For the Kafka host, we have have defined either the KAFKA_HOST environment variable with a fallback of localhost and a port set to the KAFKA_PORT environment variable with a fallback to 9092. We have also set a default topic to chat-messages which we created earlier.

⚠️ NOTE
If you want to run a quick compile from here there are a couple of things we need to add to our test configuration:

src/test/resources/application.properties
quarkus.application.name=test-websockets
quarkus.log.category."com.brightfield.streams".level=ALL
quarkus.kafka-streams.topics=chat-messages

Create the Kafka Consumer

In order to do this, we will be updating our SocketEndpoint class

First, let's create a method to broadcast to all users who are connected:

private void broadcast(String message) {
    socketSessions.values().forEach(s -> {
        s.getAsyncRemote().sendText(message, result -> {
            if (result.getException() != null) {
                log.error("Unable to send message: {}", result.getException().getMessage(), result.getException());
            }
        });
    });
}

As you can see we are iterating through the Map we created of the different user sessions indexed by user name and creating an Async Remote to send the text message to each user.

Next let's add the consumer, again in the SocketEndpoint class we want to add the following code:

@Produces
public Topology buildTopology() {
    log.info("Building the Topology...");
    StreamsBuilder builder = new StreamsBuilder();

    builder.stream("chat-messages", Consumed.with(Serdes.String(), Serdes.String()))
        .peek((id, message) -> {
            log.info("Incoming transaction: {}", message);
            broadcast(message);
        });
    return builder.build();
}

Here we have specified the stream we want to listen to and use a String KeySerializer and a String ValueSerializer to read the message from the topic. We then log the message and broadcast it to all users connected over the WebSocket.

Updating the Unit Tests

If we try and build the service we will hit a wall when running the tests if you don't have a Kafka server running. If you do, you will find the unit tests getting stuck because there is no shutdown process in the test. This is where testcontainers come into play.

In the unit test we created in the previous article, we are going to enhance it to use a new lifecycle for our test Kafka server.

First we will create our test kafka instance:

src/test/java/com/brightfield/streams/InfrastructureTestResource.java

package com.brightfield.streams;

import io.quarkus.test.common.QuarkusTestResourceLifecycleManager;
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.testcontainers.containers.KafkaContainer;

import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Collectors;

public class InfrastructureTestResource implements QuarkusTestResourceLifecycleManager {

    private final Logger log = LoggerFactory.getLogger(InfrastructureTestResource.class);
    private final KafkaContainer kafkaContainer = new KafkaContainer("5.5.1");

    @Override
    public int order() {
        return 1;
    }

    @Override
    public void init(Map<String, String> initArgs) {
        log.info("Initialising...");
    }

    @Override
    public Map<String, String> start() {
        log.info("Starting kafka test container...");
        this.kafkaContainer.start();

        log.info("Creating topic...");
        createTopics("chat-messages");
        return configurationParameters();
    }

    @Override
    public void stop() {
        this.kafkaContainer.close();
    }

    private void createTopics(String... topics) {
        var newTopics =
            Arrays.stream(topics)
                .map(topic -> new NewTopic(topic, 1, (short) 1))
                .collect(Collectors.toList());
        try (var admin = AdminClient.create(Map.of("bootstrap.servers", getKafkaBrokers()))) {
            admin.createTopics(newTopics);
        }
    }

    private String getKafkaBrokers() {
        this.kafkaContainer.getFirstMappedPort();
        return String.format("%s:%d", kafkaContainer.getContainerIpAddress(), kafkaContainer.getMappedPort(KafkaContainer.KAFKA_PORT));
    }

    private Map<String, String> configurationParameters() {
        log.info("Returning configurationParameters...");
        final Map<String, String> conf = new HashMap<>();
        String bootstrapServers = getKafkaBrokers();
        log.info("Brokers: {}", bootstrapServers);
        conf.put("bootstrap.servers", kafkaContainer.getBootstrapServers());
        conf.put("quarkus.kafka-streams.bootstrap-servers", bootstrapServers);
        conf.put("mp.messaging.outgoing.delivery.bootstrap.servers", bootstrapServers);
        return conf;
    }
}

Next we want our test to use this Resource:

src/test/java/com/brightfield/streams/SocketEndpointTest.java

@QuarkusTest
@QuarkusTestResource(value = InfrastructureTestResource.class)
public class SocketEndpointTest {
...
}

When you compile and run the unit tests, you should now see the test running the websocket tests and connecting to the kafka container and then disconnecting and not getting stuck. By creating the InfrastructureTestResource, we have basically added a lifecycle to how the Kafka container is being managed.

First, the init() method is called. In our scenario we are just logging out that the init() method has been called.
Next the start() method is called which creates the topics on the testcontainer we want to use then returns the configuration of the Kafka container.
When the tests are complete, the close() method is called to clean up and shut down the Kafka container.

⚠️
We won't be implementing full testing of the Kafka integration in this article, what we have for now will be sufficient to test from the command line.

Running our Stack

Everything should now be in place. Let's start our service and angular client application and see if it works!

Sending some test messages through the web interface should function as it did before:

To test our broadcast capabilities, we will revert to the command line and publish the messages from there.

Access the KAFKA_HOME directory in a terminal window and enter:

$ bin/kafka-console-producer.sh --broker-list=localhost:9092 --topic chat-messages
>Below
>People

You should see the user interface update with the same values:

Conclusion

With this as your base you can come up with a straight forward fully blown messaging tool; listing users, their statuses, and even group messages.

In a future article we will explore how to test the Kafka component in the meantime, happy chatting!

Quickest WebSockets with Quarkus and Angular 10

Anthony Ikeda — Mon, 07 Sep 2020 16:40:34 +0000

Quarkus helps you get started with both messaging and WebSockets really quickly. But what happens when you want to marry the two technologies together?

Before we start though the source code used in this articel can be found here:

https://github.com/cloudy-engineering/quarkus-chat-api
https://github.com/cloudy-engineering/angular-chat-ui

Getting started with Quarkus WebSockets is as simple as this:

$ mvn io.quarkus:quarkus-maven-plugin:1.7.0.Final:create \
    -DprojectGroupId=org.acme \
    -DprojectArtifactId=websockets-quickstart \
    -Dextensions="undertow-websockets"
$ cd websockets-quickstart

This will create your typical maven based source code structure with the following highlights:

quarkus-undertow-websockets dependency added
quarkus-resteasy dependency add by default

Getting your first integration up and running is actually quite easy.

Create a new class representing the WebSocket Endpoint
Implement the standard WebSocket lifecycle methods (onOpen, onError, onMessage, onClose)
Create a UI to integrate with your WebSocket endpoint

Implementing the Functionality

Rather than implementing an interface or extending a base class, Quarkus WebSockets uses annotations to implement the lifecycle:

@ServerEndpoint("/chat/{username}")
public class SocketEndpoint {

    private final Logger log = LoggerFactory.getLogger(SocketEndpoint.class);

    @OnOpen
    public void onOpen(Session session, @PathParam("username") String username) {
        log.debug("{} has just connected", username);
    }

    @OnError
    public void onError(Session session, @PathParam("username") String username, Throwable throwable) {
        log.error("{} encountered an error", username);
    }

    @OnMessage
    public void onMessage(String message, @PathParam("username") String username) {
        log.debug("{} has just sent us a message: {}", username, message);
    }

    @OnClose
    public void onClose(Session session, @PathParam("username") String username) {
        log.debug("{} has now disconnected", username);
    }
}

The thing to remember with the server-side component is the Session. This is the way you communicate with the end user. For the sake of this article we are going to use the AsyncRemote and send an object back to the user.

@ServerEndpoint("/chat/{username}")
@ApplicationScoped
public class SocketEndpoint {

    @OnOpen
    public void onOpen(Session session, @PathParam("username") String username) {
        log.debug("{} has just connected", username);
        session.getAsyncRemote().sendText(String.format("Welcome to the show %s", username));
    }

...
}

When you connect via the front end, the onOpen method will be instantiated. Here you can set up the user's interaction and send back a message confirming any actions. Here we will just send back a reply.

Logging and a CORS
Before we continue we are going to configure logging to ensure we can see our debug messages.

In the src/main/resources/application.properties file add the following entries:

quarkus.log.category."com.brightfield.streams".level=ALL

We should also enable CORS so we will also need:

quarkus.http.cors.enabled=true
quarkus.http.cors.origins=http://localhost:4200
quarkus.http.cors.methods=get,post,put,head,options

And since I have so many apps running on port 8080 I’m going to change the port to 8011:

quarkus.http.port=8011

Let's create a unit test to test this out:

ServerEndpointTest.java

package com.brightfield.streams;

import io.quarkus.test.common.http.TestHTTPResource;
import io.quarkus.test.junit.QuarkusTest;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.websocket.*;
import java.net.URI;
import java.util.concurrent.LinkedBlockingDeque;
import java.util.concurrent.TimeUnit;

@QuarkusTest
public class SocketEndpointTest {

    private static final LinkedBlockingDeque<String> MESSAGES = new LinkedBlockingDeque<>();

    @TestHTTPResource("/chat/testuser")
    URI uri;

    @Test
    public void testWebSocketChat() throws Exception {
        try (Session session = ContainerProvider.getWebSocketContainer().connectToServer(Client.class, uri)) {
            Assertions.assertEquals("Connecting to central control...", MESSAGES.poll(10, TimeUnit.SECONDS));
            Assertions.assertEquals("Welcome to the show testuser", MESSAGES.poll(10, TimeUnit.SECONDS));
        }
    }

    @ClientEndpoint
    public static class Client {
        private final Logger log = LoggerFactory.getLogger(Client.class);

        @OnOpen
        public void open(final Session session) {
            log.debug("Connecting to server");
            String toSend = "Connecting to central control...";
            session.getAsyncRemote().sendText(toSend);
        }

        @OnMessage
        void message(final String message) {
            log.debug("Incoming message: {}", message);
            MESSAGES.add(message);
        }
    }
}

So, what have we done here?

First we set up a queue to store messages that are going through the pipeline. When ever we send a message or receive it on the client side we want to queue up the messages to validate the order they arrive.

In this case the first message will be the message sent when the Client.class first connects: "Welcome to the show "

When the client connects we are going to send our first message: "Connection to central control...". This will be our second message in the sequence.

If you compile and run the code you should see our test passing with the debug much like this:

INFO  [io.und.web.jsr] (main) UT026004: Adding annotated client endpoint class com.brightfield.streams.SocketEndpointTest$Client
INFO  [io.und.web.jsr] (main) UT026003: Adding annotated server endpoint class com.brightfield.streams.SocketEndpoint for path /chat/{username}
INFO  [io.quarkus] (main) Quarkus 1.7.2.Final on JVM started in 1.791s. Listening on: http://0.0.0.0:8081
INFO  [io.quarkus] (main) Profile test activated.
INFO  [io.quarkus] (main) Installed features: [cdi, resteasy, servlet, undertow-websockets]
DEBUG [com.bri.str.SocketEndpointTest$Client] (main) Connecting to server
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-2) testuser has just connected
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-2) testuser has just sent us a message: Connecting to central control...
DEBUG [com.bri.str.SocketEndpointTest$Client] (nioEventLoopGroup-2-1) Incoming message: Welcome to the show testuser

If we consider the sequence of events:

Client connects to the Server
Server sends a welcome message: "Welcome to the show testuser"
Client sends message to the server: "Connecting to central control..."
Client receives message: "Welcome to the show testuser"

Our test tracks the server side interaction and the client side interaction.

Creating the UI

Let's look at creating a UI in Angular 10 to get a better picture.

Start by creating your Angular app:

$ ng new chat-ui
? Would you like to add Angular routing? Yes
? Which stylesheet format would you like to use? CSS
... 
Installing packages...
✔ Packages installed successfully.
    Successfully initialized git.

Next we want to make sure we have Reactive Forms lodes in the app.module.ts:

@NgModule({
...
  imports: [
    BrowserModule,
    AppRoutingModule,
    BrowserAnimationsModule,
    ReactiveFormsModule,
  ],

});

We want to create 2 classes:

Service that manages the WebSocket state
Component that displays our interactions

$ ng g s _services/socket
CREATE src/app/_services/socket.service.spec.ts (357 bytes)
CREATE src/app/_services/socket.service.ts (135 bytes)
$ ng g c chat
CREATE src/app/chat/chat.component.css (0 bytes)
CREATE src/app/chat/chat.component.html (19 bytes)
CREATE src/app/chat/chat.component.spec.ts (612 bytes)
CREATE src/app/chat/chat.component.ts (267 bytes)
UPDATE src/app/app.module.ts (388 bytes)

For the sake of best practices, let's first configure an environment variable for the Quarkus application endpoint:

src/environments/environment.ts

export const environment = {
  production: false,
  socket_endpoint: 'ws://localhost:8011'
};

Implementing our service

To connect and interact with the server-side components we are going to utilise some built-in rxjs classes:

import { webSocket, WebSocketSubject } from 'rxjs/webSocket';

RXJS and WebSockets
RXJS makes it just as easy to connect to a ServerEndpoint as it is to implement.

The WebSocketSubject represents the state of the communication between the client and server. Just like a BehaviorSubject we are going to push messages and subscribe to the response over the WebSocketSubject.

The webSocket class represents our factory to create WebSocketSubject connection to the server. We will pass in the URL to our service and it will return the WebSocketSubject for us to push and subscribe to.

There are 3 parts of the lifecycle we need to implement:

Connect (onOpen)
Close/Destroy (onClose)
Send (onMessage)

chat.service.ts

import { Injectable } from '@angular/core';
import { webSocket, WebSocketSubject } from 'rxjs/webSocket';
import { Observable } from 'rxjs';
import { environment as env } from '../../environments/environment';

@Injectable({
  providedIn: 'root'
})
export class SocketService {

  connection$: WebSocketSubject<any>;

  constructor() { }

  connect(): Observable<any> {
    this.connection$ = webSocket({
      url: `${env.socket_endpoint}/chat/angularuser`,
      deserializer: ({data}) => data,
      serializer: ({data}) => data,
    });
    return this.connection$;
  }
...
}

When creating a WebSocket connection with RXJS, the default serialization/deserialization is JSON.parse. Since we are using plain test on our server side component, we will override the serde without parsing the data.

Later we will see how we call the connect() method and make the initial connection that we can send and receive messages over.

In order to send a message, we need to queue up the message much like any Observable you may have had experience with:

import { Injectable } from '@angular/core';
import { webSocket, WebSocketSubject } from 'rxjs/webSocket';
import { Observable } from 'rxjs';
import { environment as env } from '../../environments/environment';

@Injectable({
  providedIn: 'root'
})
export class SocketService {

  connection$: WebSocketSubject<any>;

  constructor() { }

  connect(): Observable<any> {
    this.connection$ = webSocket(`${env.socket_endpoint}/angularuser`);
    return this.connection$;
  }

  send(data: any): void {
    if (this.connection$) {
      this.connection$.next(data);
    } else {
      console.log('Did not send data, unable to open connection');
    }
  }

}

While our connection$ pipe is open, we use the next() method to send our object to the server. If we have lost connectivity we will, for now, just log a message.

One last thing, if we ever disconnect from the server, we want to ensure we close the connection and trigger the backend event @OnClose, so let's implement a closeConnection() method and call it in an onDestroy() event:

chat.service.ts

import { Injectable } from '@angular/core';
import { webSocket, WebSocketSubject } from 'rxjs/webSocket';
import { Observable } from 'rxjs';
import { environment as env } from '../../environments/environment';

@Injectable({
  providedIn: 'root'
})
export class SocketService {

  connection$: WebSocketSubject<any>;

  constructor() { }

  connect(): Observable<any> {
    this.connection$ = webSocket(`${env.socket_endpoint}/angularuser`);
    return this.connection$;
  }

  send(data: any): void {
    if (this.connection$) {
      this.connection$.next(data);
    } else {
      console.log('Did not send data, unable to open connection');
    }
  }

  closeConnection(): void {
    if (this.connection$) {
      this.connection$.complete();
      this.connection$= null;
    }
  }

  ngOnDestroy() {
    this.closeConnection();
  }

}

Creating our Angular Component

As you can see we have a very straight forward implementation that uses standard Observable patterns. In order to use this service you need to create a component that will initiate the connection and send the data over the websocket connection:

import { Component, OnInit } from '@angular/core';
import { SocketService } from '../_services/socket.service';
import { Subject } from 'rxjs';
import { takeUntil } from 'rxjs/operators';
import { FormControl } from '@angular/forms';

@Component({
  selector: 'app-chat',
  templateUrl: './chat.component.html',
  styleUrls: ['./chat.component.css']
})
export class ChatComponent implements OnInit {

  messages: string[] = [];
  msgControl = new FormControl('');
  destroyed$ = new Subject();

  constructor(private chatService: SocketService) { }

  ngOnInit(): void {
    const chatSub$ = this.chatService.connect().pipe(
      takeUntil(this.destroyed$),
    );

    chatSub$.subscribe(message => this.messages.push(message));
  }

  sendMessage(): void {
    this.chatService.send(this.msgControl.value);
    this.msgControl.setValue('');
  }

  ngOnDestroy(): void {
    this.destroyed$.next();
  }

}

chat.component.html

<ul>
  <li *ngFor="let message of messages">{{ message }}</li>
</ul>
<input placeholder="Send a message..." [formControl]="msgControl">
<button (click)="sendMessage()">Send</button>

Let's also quickly add a route for our new component:

app-routing.module.ts

import { NgModule } from '@angular/core';
import { Routes, RouterModule } from '@angular/router';
import { ChatComponent } from './chat/chat.component';

const routes: Routes = [
  { path: 'chat', component: ChatComponent }
];

@NgModule({
  imports: [RouterModule.forRoot(routes)],
  exports: [RouterModule]
})
export class AppRoutingModule { }

As you can see in our component we are going to call the methods of the SocketService and run the lifecycle of the websocket connection. The user interface is a simple form control that has a list of messages that come back.

Start up the service and the angular user interface and you should be able to access the configured route at http://localhost:4200/chat

When you access the page you should see our initial message "Welcome to the show angularuser" and an input box.

If we check out the logs we should see the initial connection being made:

__  ____  __  _____   ___  __ ____  ______
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
WARN  [io.qua.kub.dep.KubernetesProcessor] (build-15) No registry was set for the container image, so 'ImagePullPolicy' is being force-set to 'IfNotPresent'.
INFO  [io.und.web.jsr] (Quarkus Main Thread) UT026003: Adding annotated server endpoint class com.brightfield.streams.SocketEndpoint for path /chat/{username}
INFO  [io.quarkus] (Quarkus Main Thread) chat-service 1.0-SNAPSHOT on JVM (powered by Quarkus 1.7.2.Final) started in 3.055s. Listening on: http://0.0.0.0:8011
INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, kubernetes, resteasy, servlet, undertow-websockets]
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-18) angularuser has just connected

If you enter a message and hit Send, you should see the message being logged server-side:

__  ____  __  _____   ___  __ ____  ______
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
WARN  [io.qua.kub.dep.KubernetesProcessor] (build-15) No registry was set for the container image, so 'ImagePullPolicy' is being force-set to 'IfNotPresent'.
INFO  [io.und.web.jsr] (Quarkus Main Thread) UT026003: Adding annotated server endpoint class com.brightfield.streams.SocketEndpoint for path /chat/{username}
INFO  [io.quarkus] (Quarkus Main Thread) chat-service 1.0-SNAPSHOT on JVM (powered by Quarkus 1.7.2.Final) started in 3.055s. Listening on: http://0.0.0.0:8011
INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, kubernetes, resteasy, servlet, undertow-websockets]
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-18) angularuser has just connected
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-18) angularuser has just sent us a message: "Good morning"
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-8) angularuser has just connected

So far so good, but we want this to be more interactive. For the sake of this article, let's just echo back what the user sends:

@ServerEndpoint("/chat/{username}")
public class SocketEndpoint {

    private final Logger log = LoggerFactory.getLogger(SocketEndpoint.class);
    private Map<String, Session> socketSessions = new HashMap<>();

    @OnOpen
    public void onOpen(Session session, @PathParam("username") String username) {
        log.debug("{} has just connected", username);
        session.getAsyncRemote().sendText(String.format("Welcome to the show %s", username));
        socketSessions.put(username, session);
    }

    @OnError
    public void onError(Session session, @PathParam("username") String username, Throwable throwable) {
        log.error("{} encountered an error", username);
    }

    @OnMessage
    public void onMessage(String message, @PathParam("username") String username) {
        log.debug("{} has just sent us a message: {}", username, message);
        Session session = socketSessions.get(username);
        session.getAsyncRemote().sendText(message);
    }

    public void onClose(Session session, @PathParam("username") String username) {
        log.debug("{} has now disconnected", username);
    }
}

In the update to the code, when the user connects, we are going to keep a reference of the Session in a HashMap indexed on the username. When a message comes in, we will lookup the session and then send the message back.

DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-12) angularuser has just connected
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-12) angularuser has just sent us a message: "Glad to be here"
DEBUG [com.bri.str.SocketEndpoint] (vert.x-eventloop-thread-12) angularuser has just sent us a message: "What day is it?"

In the next article I will demonstrate how to wire up Kafka to the WebSocket sessions and broadcast messages coming in from the Kafka queues.

GitHub Actions, Maven & Artifactory

Anthony Ikeda — Wed, 02 Sep 2020 16:07:06 +0000

I recently started a new Proof of Concept that started with a single repo in GitHub and built using GitHub Actions.

Then I externalised some base classes to share across a couple of repositories and, of course, broke the build because the shared classes were no longer immediately available.

This started my journey to setting up an Artifactory instance and performing proper releases on my Proof of Concept and I wanted to share how to get set up with:

Maven
Artifactory
GitHub Actions

Getting Started

Of course you will need to have a couple of GitHub Repositories and an Artifactory instance set up. The free version of Artifactory works perfectly for this scenario thought remember, there may be other steps required to setup a different dependency management tool.

First let's look at how our system authentication looks:

In order to manage the artifacts (code, jars, bill of materials) we will need to be able to use our Artifactory instance as one of our Maven repositories. This requires ensuring we can authenticate to read from our repository.

We also will need to ensure we have permissions to push to the Artifactory instace.

Lastly in order to tag and push a release in GitHub, we will need to configure Maven with the correct security credentials to do so on your behalf.

As a reference here are the different security variables we need to set up:

Environment var	Secret Value	Description
ARTIFACTORY_USERNAME_REF	'stream-github-actions'	It's not actually used but more a placeholder since we will be using the Authorization Header with the token
ARTIFACTORY_TOKEN_REF	secrets.ARTIFACTORY_TOKEN	The bearer token that has the permissions to read and write to the selected Artifactory repositories.
ARTIFACTORY_TOKEN	secrets.ARTIFACTORY_TOKEN	This token is used by the Artifactory maven plugin and uses the same token that is used to read from Artifactory.
SCM_USERNAME_REF	github-actions	Just a reference point for a committer
SCM_PASSWORD_REF	secrets.GITHUB_TOKEN	Enables controls on the git code for checkout, commit and tagging.

Creating the Artifactory Auth Tokens

There are 2 steps to setting up the Artifactory Auth tokens:

Creating a group and permissions to access repositories
Using the command line to generate the tokens

In your newly created Artifactory instance we want to first create a group that we can assign permissions to.

Access Artifactory and navigate to: Administration → Identity & Access → Groups and click "Create new group"
Give the group a name and click "Save"

Next, we want to create a permission scheme to allocate to the group.

Access Administration → Identity & Access → Permissions and click 'New Permission'
Give the permission a name (e.g. tutorial-cicd)
Click Add Repositories and select the repositories you want to grant access to (libs-release-local, libs-snapshot-local) and click Save
Click the plus sign on the Builds section then select 'Any'
Next click on the Groups tab and click the plus sign
Drag the actions-cicd group to the right and hit Ok.
Now we want to select the Repository permissions so choose: Read, Deploy/Cache
Next we want to allow build permissions so select: Read, Deploy
Click 'Create' to return to the Permission screen
Click 'Save'

Generating the Artifactory Auth Tokens

There is no way to set up auth tokens in the Artifactory UI so we will use the command line to generate them.

First let's set up some environment variables in the terminal:



$ export ART_USERNAME= #name used to log in to your artifactory
$ export ART_PASSWORD= # password to log in to your artifactory
$ export ART_GROUP=tutorial-cicd
export ART_HOSTNAME= # hostname of your artifactory instance

Next let's call your Artifactory API to generate the token:



$ curl -u "${ART_USERNAME}:${ART_PASSWORD}" -XPOST "https://${ART_HOSTNAME}/artifactory/api/security/token" \
  -d "username=stream-github-actions" \
  -d scope="member-of-groups:${ART_GROUPNAME} api:*" \
  -d expires_in=0

The main thing to pay attention to here is the scope:

scope="member-of-groups:${ART_GROUPNAME} api:*"

the member-of-groups ensures that the group we just set up with permissions is allocated to the user stream-github-actions ensuring the token grants the permissions we need.

You'll also note that we have set expires_in=0 which ensures the token does not expire.

The response should be similar to:



{
  "scope" : "member-of-groups:actions_cicd api:*",
  "access_token" : "xxxxxxxxx",
  "expires_in" : 0,
  "token_type" : "Bearer"
}

Make sure you keep the username and token handy as once we add it to GitHub Secrets you'll never see it again!

Creating the GitHub Secrets

Next we have to make sure we can use these values and since this is sensitive data, we will make use of GitHub Secrets to manage them.

Access the GitHub repository for your build and navigate to Settings → Secrets.

Click 'New Secret' and create the following secret:

Secret Name	Secret Value
ARTIFACTORY_TOKEN	The value returned

Maven POM Updates

Time to move on to configuring you maven project!

What needs to be configured:

Maven Profile
- Used to ensure we utilise the right repositories when we build the project in GitHub Actions
SCM setup
- This defines where the code exists so that when we perform a release, maven can connect to the GitHub repo.
Repository credentials for both Artifactory and GitHub in the Maven settings.xml file
The maven-artifactory-plugin
The maven-release-plugin

Set up a Maven Profile

Maven profiles are handy for configuring how you manage dependencies based on certain parameters. You can, for example, have a local setup that pulls from the default repositories you need. You can also ad a repository to your local setup with your own private credentials. However, this file (settings.xml) is not portable.

We will setup the repositories in the project pom.xml and have it activate only when an environment variable (BUILD_ENV) is configured.

In your project pom.xml file we want to first add the following profile:



<profiles>
  <profile>
    <id>github-actions</id>
    <activation>
      <property>
        <name>env.BUILD_ENV</name>
        <value>github-actions</value>
      </property>
    </activation>
    <repositories>
      <repository>
        <id>artifactory</id>
        <name>your-artifactory</name>
        <url>https://your-artifactory.jfrog.io/artifactory/libs-release-local</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </repository>
    </repositories>
    <pluginRepositories>
      <pluginRepository>
        <id>artifactory</id>
        <name>your-artifactory</name>
        <url>https://your-artifactory.jfrog.io/artifactory/libs-release-local</url>
        <releases>
          <enabled>true</enabled>
        </releases>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
      </pluginRepository>
      <pluginRepository>
        <snapshots>
          <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray-plugins</name>
        <url>https://jcenter.bintray.com</url>
      </pluginRepository>
    </pluginRepositories>
  </profile>
</profiles>

So what have we just done?

Above we have created a new profile that will only be active if you have set the BUILD_ENV environment variable to github-actions - we will manage this in our GitHub Actions pipeline.

Next we've defined the repositories to pull dependencies from. Artifactory acts as a proxy for artifacts that are considered standard access. However, it will now search your customer Artifactory for any dependencies you have already released and stored.

Finally we still need to load certain plugins that originate from the BinTray repository (maven-artifactory-plugin) which is why we have added in the the BinTray Plugin repository.

Credentials for you repository will be configured outside the pom.xml in a custom settings.xml file we will set up in our build pipeline.

Plugin Setup

As mentioned there are 2 plugins we need to configure:

The Maven release plugin will enable the release:prepare and the release:perform actions.

In your pom.xml update the <build> section with:



<build>
  <plugins>
    <plugin>
        <artifactId>maven-release-plugin</artifactId>
        <version>3.0.0-M1</version>
        <configuration>
          <tagNameFormat>v@{project.version}</tagNameFormat>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.jfrog.buildinfo</groupId>
        <artifactId>artifactory-maven-plugin</artifactId>
        <version>2.7.0</version>
        <inherited>false</inherited>
        <executions>
          <execution>
            <id>build-info</id>
            <goals>
              <goal>publish</goal>
            </goals>
            <configuration>
              <publisher>
                <contextUrl>https://your-artifactory.jfrog.io/artifactory</contextUrl>
                <username>stream-github-actions</username>
                <password>${env.ARTIFACTORY_TOKEN}</password>
                <repoKey>libs-release-local</repoKey>
                <snapshotRepoKey>libs-snapshot-local</snapshotRepoKey>
              </publisher>
            </configuration>
          </execution>
        </executions>
      </plugin>
  </plugins>
</build>

You can see in the Artifactory plugin, we are passing the token we generated as the password in via the environment variable. This will only be used for when we push the jar file to our Artifactory instance.

SCM Setup

So that we can perform versioned releases, we next need to configure the SCM provider.

In the pom.xml you want to add the following somewhere near the top of the file:



  <scm>
    <developerConnection>scm:git:https://github.com/your-org/your-repo.git/</developerConnection>
    <url>https://github.com/your-org/your-repo</url>
    <tag>HEAD</tag>
  </scm>

Here we have specified the type of scm provider (scm:git) and the location of the repository. We also want to make sure when we release that maven will tag the HEAD of the branch with the new version.

One last setup item is going to be to configure our server credentials:



<properties>
  ...
  <project.scm.id>github</project.scm.id>
</properties>

This indicates the server configuration that we are going to use in the settings.xml file we will generate in GitHub actions and ensures that we can successfully tag and bump the project version.

Creating our Build Pipeline

Okay it's been a long journey so far but we are close to the finish line. We are now going to take what we've created and build out our build and release pipeline.

What we will be using

First let me outline what we will be doing:

Create a standard GitHub Actions Maven build pipeline
Customizing 2 jobs:
- Build and Unit Test
- Prepare and Perform release

The Build and Unit test job will run on all branches and pull requests, however, our release will only run on the master branch.

Basic Build

Go to your GitHub repository and select Actions and then select Java with Maven - Set up this workflow:

For now, let's just get the build going with the basic setup so let's make some small changes:



name: Java CI with Maven

on:
  push:
    branches: [ "**" ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 11.0.2
        uses: actions/setup-java@v1
        with:
          java-version: 11.0.2
      - name: Build with Maven
        run: mvn -B package --file pom.xml

The first change we will make is to ensure the build runs on all branches, so you can change the on.push.branches to ["**"]

Next I've configured the JDK to build with by setting the java-version to 11.0.2.

For now commit the file and the very first build should run.

Now we will setup the settings.xml file.

Setting up settings

Why do we need this file? Well as mentioned earlier, we need to store credentials somewhere, in this case we will be referencing environment variables to ensure we don't expose any sensitive data.

The action we will use to create the file is the whelk-io/maven-settings-xml-action@v11 This will allow us to create a custom built settings.xml file that maven will use:



name: Java CI with Maven

on:
  push:
    branches: [ "**" ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 11.0.2
        uses: actions/setup-java@v1
        with:
          java-version: 11.0.2
      - name: Setup Maven settings.xml
        uses: whelk-io/maven-settings-xml-action@v11
        with:
          servers: |
            '[
              {
                "id": "artifactory",
                "username": "${env.ARTIFACTORY_USERNAME_REF}",
                "password": "${env.ARTIFACTORY_TOKEN_REF}"
              }
            ]'
     - name: Build with Maven
       env:
          ARTIFACTORY_USERNAME_REF: "stream-github-actions"
          ARTIFACTORY_TOKEN_REF: ${{ secrets.ARTIFACTORY_TOKEN }}
          BUILD_ENV: 'github-actions'
       run: mvn -B package --file pom.xml

As you can see we aren't embedding any sensitive date, just environment variables. When this step runs a newly created settings.xml file will exist with the following content:



<settings>
  <servers>
    <server>
      <id>artifactory</id>
      <username>${env.ARTIFACTORY_USERNAME_REF}</username>
      <password>${env.ARTIFACTORY_TOKEN_REF}</password>
    </server>
  </servers>
</settings>

Now when the build runs, we want to use the injected secrets as these environment variables.

You'll also note we are setting the BUILD_ENV environment variable so that Maven uses the profile we set up with pull artifacts from our own Artifactory server.

You can also see in the maven.yml build file that we are mapping our secrets and values to the environment variables that our pom.xml and settings.xml files are using.

Commit the changes and your build should run and pull the dependencies from your own Artifactory instance.

Doing a release

Now that we have a build running successfully we can apply a release and version our artifact. As mentioned earlier, we will make sure we bump the artifact version and push to Artifactory only when a merge to master is successful.

Most of the building blocks are in place right now and we just need to add a release stage to our main.yml actions file:



name: Java CI with Maven

on:
  push:
    branches: [ "**" ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - name: Set up JDK 11.0.2
        uses: actions/setup-java@v1
        with:
          java-version: 11.0.2
      - name: Setup Maven settings.xml
        uses: whelk-io/maven-settings-xml-action@v11
        with:
          servers: |
            '[
              {
                "id": "artifactory",
                "username": "${env.ARTIFACTORY_USERNAME_REF}",
                "password": "${env.ARTIFACTORY_TOKEN_REF}"
              }
            ]'
     - name: Build with Maven
       env:
          ARTIFACTORY_USERNAME_REF: "stream-github-actions"
          ARTIFACTORY_TOKEN_REF: ${{ secrets.ARTIFACTORY_TOKEN }}
          BUILD_ENV: 'github-actions'
       run: mvn -B package --file pom.xml

  release:
    runs-on: ubuntu-latest
    needs: build
    if: github.ref == 'refs/heads/master'

    steps:
      - uses: actions/checkout@v2
      - name: Setup JDK 11
        uses: actions/setup-java@v1.3.0
        with:
          java-version: 11.0.2

      - name: maven-settings-xml-action
        uses: whelk-io/maven-settings-xml-action@v11
        with:
          servers: '
            [
              {
                "id": "github",
                "username": "${env.SCM_USERNAME_REF}",
                "password": "${env.SCM_TOKEN_REF}"
              },
              {
                "id": "artifactory",
                "username": "${env.ARTIFACTORY_USERNAME_REF}",
                "password": "${env.ARTIFACTORY_TOKEN_REF}"
              }
            ]'
      - name: Configure git
        run: |
          git config --global committer.email "noreply@github.com"
          git config --global committer.name "GitHub"
          git config --global author.email "${GITHUB_ACTOR}@users.noreply.github.com"
          git config --global author.name "${GITHUB_ACTOR}"
      - name: Prepare the release
        env:
          SCM_USERNAME_REF: 'github-actions'
          SCM_TOKEN_REF: ${{ secrets.GITHUB_TOKEN }}
          ARTIFACTORY_USERNAME_REF: "stream-github-actions"
          ARTIFACTORY_TOKEN_REF: ${{ secrets.ARTIFACTORY_TOKEN }}
          BUILD_ENV: 'github-actions'
        run: |
          ./mvnw --batch-mode release:prepare
      - name: Push the release
        env:
          SCM_USERNAME_REF: 'github-actions'
          SCM_TOKEN_REF: ${{ secrets.GITHUB_TOKEN }}
          ARTIFACTORY_USERNAME_REF: "stream-github-actions"
          ARTIFACTORY_TOKEN_REF: ${{ secrets.ARTIFACTORY_TOKEN }}
          ARTIFACTORY_TOKEN:  ${{ secrets.ARTIFACTORY_TOKEN }}
          BUILD_ENV: 'github-actions'
        run: ./mvnw --batch-mode release:perform
      - name: Rollback if failure
        if: ${{ failure() }}
        run: ./mvnw --batch-mode release:rollback

There are 6 steps we are introducing here:

Setup our JDK we want to build with
Create our settings.xml file but this time with a new server entry for GitHub
Configure a Git username for commits
The Prepare release step
The Perform release step
If the release process fails, we want to roll back the steps that set up the release.

To prevent the release job from running before we've successfully built and unit tested the project, we've included a needs: build directive. This ensures that unless the build job passes, the release job won't run.

We are also validating that we are only running the release on the master branch as indicated by if: github.ref == 'refs/heads/master'

In order to authenticate GitHub, we've added a new server to our settings.xml file. If you recall we added a property to the pom.xml file <project.scm.id>github</project.scm.id> which tells maven to use the GitHub server configuration with the id of 'github'.

Also note we are using reserved secret for the credentials for GitHub: secret.GITHUB_TOKEN. This is a reserved secret and enables you to use it for any actions you need to perform on the repository.

Once you've committed your work to master, the build should run, your release will run and you will find a new version tag and the artifact uploaded to your Artifactory instance.