How to quickly test connectivity to your Azure Event Hubs for Kafka cluster, without writing any code

#azure #productivity #showdev

You have a shiny new Kafka enabled broker on Azure Event Hubs and want to quickly test it out without writing cumbersome client (producer and consumer) code. Try out the instructions in this post and you should (hopefully) have everything setup and sanity tested in ~ 10 minutes.

Clone this GitHub repo and change to the correct directory

git clone https://github.com/abhirockzz/azure-eventhubs-kafka-cli-quickstart
cd azure-eventhubs-kafka-cli-quickstart

You can either choose to install the Azure CLI if you don't have it already (should be quick!) or just use the Azure Cloud Shell from your browser.

Create your Kafka enabled Event Hubs cluster

If you have a cluster already, skip this and go to the "Get what you need to connect to the cluster" section

Set some variables to avoid repetition

AZURE_SUBSCRIPTION=[to be filled]
AZURE_RESOURCE_GROUP=[to be filled]
AZURE_LOCATION=[to be filled]
EVENT_HUBS_NAMESPACE=[to be filled]
EVENT_HUB_NAME=[to be filled]

Create the resource group if you don't have one already

az account set --subscription $AZURE_SUBSCRIPTION
az group create --name $AZURE_RESOURCE_GROUP --location $AZURE_LOCATION

Create an Event Hubs namespace (similar to a Kafka Cluster)

az eventhubs namespace create --name $EVENT_HUBS_NAMESPACE --resource-group $AZURE_RESOURCE_GROUP --location $AZURE_LOCATION --enable-kafka true --enable-auto-inflate false

And then create an Event Hub (same as a Kafka topic)

az eventhubs eventhub create --name $EVENT_HUB_NAME --resource-group $AZURE_RESOURCE_GROUP --namespace-name $EVENT_HUBS_NAMESPACE --partition-count 3

Get what you need to connect to the cluster

Get the connection string and credentials for your cluster

For details, read how Event Hubs uses Shared Access Signatures for authorization

Start by getting the Event Hub rule/policy name

EVENT_HUB_AUTH_RULE_NAME=$(az eventhubs namespace authorization-rule list --resource-group $AZURE_RESOURCE_GROUP --namespace-name $EVENT_HUBS_NAMESPACE | jq '.[0].name' | sed "s/\"//g")

And, then make use of the rule name to extract the connection string

az eventhubs namespace authorization-rule keys list --resource-group $AZURE_RESOURCE_GROUP --namespace-name $EVENT_HUBS_NAMESPACE --name $EVENT_HUB_AUTH_RULE_NAME | jq '.primaryConnectionString'

This information is sensitive - please excercise caution

Copy the result of above command in the password field within jaas.conf file.

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="$ConnectionString"
    password=enter-connection-string-here;
};

DO NOT remove the trailing ; in jaas.conf

Connect to the cluster using Kafka CLI

I am assuming that you already have a Kafka setup (local or elsewhere) - the Kafka CLI is bundled along with it. If not, it shouldn't take too long to set it up - just download, unzip and you're ready to go!

On your local machine, use a new terminal to start a Kafka Consumer - set the required variables first

EVENT_HUBS_NAMESPACE=[to be filled]
EVENT_HUB_NAME=[to be filled]
export KAFKA_OPTS="-Djava.security.auth.login.config=jaas.conf"
KAFKA_INSTALL_HOME=[to be filled] e.g. /Users/jdoe/kafka_2.12-2.3.0/

Start consuming

$KAFKA_INSTALL_HOME/bin/kafka-console-consumer.sh --topic $EVENT_HUB_NAME --bootstrap-server $EVENT_HUBS_NAMESPACE.servicebus.windows.net:9093 --consumer.config client_common.properties

Use another terminal to start a Kafka Producer - set the required variables first

EVENT_HUBS_NAMESPACE=[to be filled]
EVENT_HUB_NAME=[to be filled]
export KAFKA_OPTS="-Djava.security.auth.login.config=jaas.conf"
KAFKA_INSTALL_HOME=[to be filled] e.g. /Users/jdoe/kafka_2.12-2.3.0/

Start the producer

$KAFKA_INSTALL_HOME/bin/kafka-console-producer.sh --topic $EVENT_HUB_NAME --broker-list $EVENT_HUBS_NAMESPACE.servicebus.windows.net:9093 --producer.config client_common.properties

You will get a prompt and you can start entering values e.g.

> foo
> bar
> baz
> john
> doe

Switch over to the consumer terminal to confirm you have got the messages!

For your reference...

Here is a handy list of all the Event Hubs related CLI commands which were used

az eventhubs namespace create - Creates the EventHubs Namespace.
az eventhubs eventhub create - Creates the EventHubs Eventhub.
az eventhubs namespace authorization-rule list - Shows the list of Authorizationrule by Namespace.
az eventhubs namespace authorization-rule keys list - Shows the connection strings for namespace.

I really hope you enjoyed and learned something from this article! Please like and follow if you did. Happy to get feedback via @abhi_tweeter or just drop a comment.