Authenticating Camel K to Kafka using SASL

Kuthumi Pepple — Thu, 04 Aug 2022 01:39:00 +0000

Apache Camel K is a lightweight integration framework built from Apache Camel that runs natively on Kubernetes and is specifically designed for serverless and microservice architectures. Often referred to as the Swiss army knife for solving integration problems, it can be used to easily integrate many heterogeneous systems and applications allowing them share data seamlessly. Camel K users can instantly run integration code written in Camel DSL on their preferred cloud (Kubernetes or OpenShift). To learn more about Camel K and Camel, visit the official Camel site

Kafka can be easily integrated with the rest of your applications using Camel K. Kafka brokers support client authentication using SASL (Simple Authentication and Security Layer). SASL decouples authentication mechanisms from application protocols thus allowing us to use any of the authentication mechanisms supported by SASL. We will use the SASL supported OAUTHBEARER mechanism for authentication and SSL will handle the data encryption.

Getting started

This demonstration requires access to a kubernetes/openshift cluster.

Installing Camel K and required CLI tools

Camel K: Follow the Camel K installation guide for your specific cluster
kamel: the Apache Camel K CLI tool. Can be downloaded from the Camel K releases page
kubectl: the kubernetes default CLI tool. Can be downloaded from the Kubernetes releases page

Setting up the Kafka instance

Using Openshift Streams for Apache Kafka, we can easily create a Kafka instance, a service account and a Kafka Topic. The SASL OAUTHBEARER authentication method should be selected. For this demonstration, we are going to name the Kafka Topic "test". Save the Kafka broker URL, service account ID, service account secret and token endpoint URL. We will use them in a configuration file.

Using Kafka in a Camel K integration

We will create the following files:

application.properties: Configuration file, holds configuration properties
SaslSSLKafkaProducer.java: Producer, contains integration code and produces to the Kafka Topic
SaslSSLKafkaConsumer.java: Consumer, contains integration code and consumes from the Kafka Topic

Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration, hence we will specify the JAAS configuration in the application.properties config file. We also have to specify a login callback handler that will retrieve the oauthbearer token.

application.properties file:

# Kafka config
camel.component.kafka.brokers = <YOUR KAFKA BROKER URL>
camel.component.kafka.security-protocol = SASL_SSL

camel.component.kafka.sasl-mechanism = OAUTHBEARER
camel.component.kafka.sasl-jaas-config = org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
        oauth.client.id='<YOUR SERVICE ACCOUNT ID>' \
        oauth.client.secret='<YOUR SERVICE ACCOUNT SECRET>' \
        oauth.token.endpoint.uri="<TOKEN ENDPOINT URL>" ;
camel.component.kafka.additional-properties[sasl.login.callback.handler.class] = io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler

consumer.topic=test
producer.topic=test

In the integration files, we will specify kafka-oauth-client as a maven dependency because it provides the login handler class specified in the configuration file.

SaslSSLKafkaProducer.java:

// camel-k: language=java dependency=mvn:org.apache.camel.quarkus:camel-quarkus-kafka dependency=mvn:io.strimzi:kafka-oauth-client:0.10.0

import org.apache.camel.builder.RouteBuilder;

public class SaslSSLKafkaProducer extends RouteBuilder {
  @Override
  public void configure() throws Exception {
    log.info("About to start route: Timer -> Kafka ");
    from("timer:foo")
        .routeId("FromTimer2Kafka")
        .setBody()
            .simple("Message #${exchangeProperty.CamelTimerCounter}")
        .to("kafka:{{producer.topic}}")
        .log("Message correctly sent to the topic!");
  }
}

SaslSSLKafkaConsumer.java:

// camel-k: language=java dependency=mvn:org.apache.camel.quarkus:camel-quarkus-kafka dependency=mvn:io.strimzi:kafka-oauth-client:0.10.0

import org.apache.camel.builder.RouteBuilder;

public class SaslSSLKafkaConsumer extends RouteBuilder {
  @Override
  public void configure() throws Exception {
    log.info("About to start route: Kafka -> Log ");
    from("kafka:{{consumer.topic}}")
        .routeId("FromKafka2Log")
        .log("${body}");
  }
}

Bundle configuration properties into a secret:

kubectl create secret generic kafka-props --from-file application.properties

Producing to the Kafka Topic

kamel run --config secret:kafka-props SaslSSLKafkaProducer.java --dev
...
[2] 2021-05-06 08:48:11,854 INFO  [FromTimer2Kafka] (Camel (camel-1) thread #1 - KafkaProducer[test]) Message correctly sent to the topic!
[2] 2021-05-06 08:48:11,854 INFO  [FromTimer2Kafka] (Camel (camel-1) thread #3 - KafkaProducer[test]) Message correctly sent to the topic!
[2] 2021-05-06 08:48:11,973 INFO  [FromTimer2Kafka] (Camel (camel-1) thread #5 - KafkaProducer[test]) Message correctly sent to the topic!

Consuming from the Kafka Topic

kamel run --config secret:kafka-props SaslSSLKafkaConsumer.java --dev
...
[1] 2021-05-06 08:51:08,991 INFO  [FromKafka2Log] (Camel (camel-1) thread #0 - KafkaConsumer[test]) Message #8
[1] 2021-05-06 08:51:10,065 INFO  [FromKafka2Log] (Camel (camel-1) thread #0 - KafkaConsumer[test]) Message #9
[1] 2021-05-06 08:51:10,991 INFO  [FromKafka2Log] (Camel (camel-1) thread #0 - KafkaConsumer[test]) Message #10

That's it! Hopefully someone finds this useful and if you have any questions or comments, feel free to post in the comments section. Happy Coding!

Outreachy Blog #1: Introduce Yourself

Kuthumi Pepple — Mon, 06 Jun 2022 09:05:40 +0000

I'm Kuthumi Pepple, I studied Computer Science at Obafemi Awolowo University, Nigeria. I'm a firm believer in the power of technology to improve lives. I like coding, gaming and learning about new technologies. Recently, I've bought into the ideas of the open-source movement and enjoy making contributions to open-source projects. I'm currently an Outreachy intern in Apache (more about this below)

My Core Values

Competence

Anything worth doing is worth doing well (Philip Stanhope)

I try my best to uphold this standard in every area of my life, especially in my career as a software developer where incompetence can be costly either in the present or future. I try never to submit work that I've judged to be subpar.

Contribution

Measure your life not by its duration but by its contribution (Peter Marshall)

This is important to me for several reasons. Firstly, making contributions helps others and makes the world a better place. Secondly, there is a gratifying feeling that comes with making contributions to either a project, community or cause. Finally, it is my way of giving back. Given that I've benefited from the contributions of others, whether its free online learning resources, free software or a mentor showing me a thing or two, it's only right that I make contributions that benefit others as well.

Pleasure

I finally figured out that the only reason to be alive is to enjoy it (Rita Mae Brown)

I don't believe in the afterlife, so I try to experience all the happiness that this life has to offer. I try to have fun or at the very least derive some mental satisfaction in most things I choose to do. Life is too short to spend it on things that are non-stimulating.

What motivated me to apply to Outreachy

I've always wanted to make some impact as a software developer so when I first learned about the Outreachy Internship Program, I applied in a heartbeat. Outreachy provides paid internships in open source to underrepresented groups in the tech industry (find out more here). Although I wasn't successful in my first attempt, I got some valuable experience which led to me being successful in a subsequent attempt.

DEV Community: Kuthumi Pepple