Azure Event Hubs "Role Based Access Control" in action

Azure Event Hubs is streaming platform and event ingestion service that can receive and process millions of events per second. In this blog, we are going to cover one of the security aspects related to Azure Event Hubs.

Shared Access Signature (SAS) is a commonly used authentication mechanism for Azure Event Hubs which can be used to enforce granular control over the type of access you want to grant - it works by configuring rules on Event Hubs resources (namespace or topic). However, it is recommended that you use Azure AD credentials (over SAS) whenever possible since it provides similar capabilities without the need to manage SAS tokens or worry about revoking a compromised SAS.

To read more about this topic, check out "Authenticate access to Event Hubs resources using shared access signatures (SAS)".

There a couple of ways in which you can do this:

• using Managed Identity or
• by using service principal in your client applications

We will explore the second option i.e. how to use Azure Active Directory based authentication in your Azure Event Hubs client applications. With a practical example, you will learn:

• Overview of Azure Event Hubs roles
• How to use Azure CLI to configure Service Principal and RBAC policies for Event Hubs
• Event Hubs supports multiple programming languages with specific SDKs and we'll see how to use RBAC with Java and Go clients

The code is available in this GitHub repo - https://github.com/abhirockzz/azure-eventhubs-rbac-example

Overview

This section provides a high level overview for you to get an understand the key terminologies: Roles, Security principals and Role based access control (RBAC).

Roles: Azure Event Hubs defines specific roles each of which allows you to take specific action on its resources - Data Owner, Data Sender, Data Receiver. They are quite self explanatory - Sender and Receiver roles only allow send and receive respectively, while the Owner role is like an admin privilege which allows you to complete access.

For details on these roles, please refer to their documentation links - Azure Event Hubs Data Owner, Azure Event Hubs Data Sender, Azure Event Hubs Data Receiver

RBAC, Service Principals: Service Principals are entities to who, these roles are granted - this is "Role Based Access Control" since the role you grant to the Service Principal defines which actions they can perform - in this case, the actions are: send, receive or everything!

... the scenario

Here is the example we will use to learn this. There are two Event Hubs client apps - a Java producer and a Go consumer. We will configure fine grained rules (i.e. enforce RBAC) such that:

• Java app can only send messages to an Event Hubs topic, and,
• Go app can only receive messages from an Event Hubs topic

Here are relevant code snippets:

In the Java producer client, the DefaultAzureCredentialBuilder is used.

String eventhubsNamespace = System.getenv("EVENTHUBS_NAMESPACE");
String eventhubName = System.getenv("EVENTHUB_NAME");

EventHubProducerClient producer = new EventHubClientBuilder().credential(eventhubsNamespace, eventhubName, new DefaultAzureCredentialBuilder().build()).buildProducerClient();

By convention (default), the Java SDK tries to read pre-defined environment variables for Service Principal info and authenticate based on that. If this does not work, it falls back to try the SAS auth mechanism and looks for another set of environment variables

Similarly, in the Go consumer client, the Event Hubs client creation process is greatly simplified as well

ehNamespace := os.Getenv("EVENTHUBS_NAMESPACE")
ehName := os.Getenv("EVENTHUB_NAME")

hub, err := eventhub.NewHubWithNamespaceNameAndEnvironment(ehNamespace, ehName)

The developer experience is uniform across SDKs and NewHubWithNamespaceNameAndEnvironment does the same thing as the DefaultAzureCredentialBuilder in terms of following a fixed convention attempting to authenticate using Azure Active Directory (AAD) backed Service Principal first, followed by SAS mechanism

With that said, you do need a few things to get through this tutorial:

Pre-requisites

You will need a Microsoft Azure account. Go ahead and sign up for a free one!

Azure CLI or Azure Cloud Shell - you can either choose to install the Azure CLI if you don't have it already (should be quick!) or just use the Azure Cloud Shell from your browser.

Let's get going.. start off by setting up Azure Event Hubs

Setup Azure Event Hubs

This is all done via Azure CLI, but you can also use the Azure Portal if you like. The end goal is to have an Azure Event Hubs namespace along with an Event Hub (aka topic)

Create an Azure resource group if you don't have one already

AZURE_SUBSCRIPTION=[to be filled]
AZURE_RESOURCE_GROUP=[to be filled]
AZURE_LOCATION=[to be filled]

az account set --subscription $AZURE_SUBSCRIPTION
az group create --name $AZURE_RESOURCE_GROUP --location $AZURE_LOCATION

Use az eventhubs namespace create to create an Event Hubs namespace

EVENT_HUBS_NAMESPACE=[to be filled]

az eventhubs namespace create --name $EVENT_HUBS_NAMESPACE --resource-group $AZURE_RESOURCE_GROUP --location $AZURE_LOCATION  --enable-auto-inflate false

And then create an Event Hub using az eventhubs eventhub create

EVENT_HUB_NAME=[to be filled]

az eventhubs eventhub create --name $EVENT_HUB_NAME --resource-group $AZURE_RESOURCE_GROUP --namespace-name $EVENT_HUBS_NAMESPACE --partition-count 3

.. now for the the key parts...

... security configuration

We will use Azure CLI to create Service Principals using az ad sp create-for-rbac

For the sender application (we name it eh-sender-sp)

az ad sp create-for-rbac -n "eh-sender-sp"

You will get a JSON response as such - please note down the appId, password and tenant

{
  "appId": "fe7280c7-5705-4789-b17f-71a472340429",
  "displayName": "eh-sender-sp",
  "name": "http://eh-sender-sp",
  "password": "29c719dd-f2b3-46de-b71c-4004fb6116ee",
  "tenant": "42f988bf-86f1-42af-91ab-2d7cd011db42"
}

For the receiver application (we name it eh-receiver-sp)

az ad sp create-for-rbac -n "eh-receiver-sp"

You will get a JSON response - please note down the appId, password and tenant

You can also use the Azure Portal to create a Service Principal and generate a Client Secret

Enforce RBAC

Going through this process using CLI involves a few steps, but it's beneficial in the long term e.g. for automation. It can also be carried out using the Azure Portal

Get the IDs for both the roles using az role definition list

EH_SENDER_ROLE_ID=$(az role definition list -n "Azure Event Hubs Data Sender" -o tsv --query '[0].id')
EH_RECEIVER_ROLE_ID=$(az role definition list -n "Azure Event Hubs Data Receiver" -o tsv --query '[0].id')

Azure Event Hubs Data Sender and Azure Event Hubs Data Receiver are the role names

Get the subscription ID for Azure Event Hubs namespace using az eventhubs namespace show

export RESOURCE_GROUP=[replace with resource group name]
export EVENTHUBS_NAMESPACE=[replace with namespace]

EVENTHUBS_ID=$(az eventhubs namespace show --resource-group $RESOURCE_GROUP --name $EVENTHUBS_NAMESPACE -o tsv --query 'id')

Assign roles using az role assignment create

export SENDER_SP_ID=[replace with Service Principal "appId" for the sender SP]
export RECEIVER_SP_ID=[replace with Service Principal "appId" for the receiver SP]

az role assignment create --assignee $SENDER_SP_ID --role $EH_SENDER_ROLE_ID --scope $EVENTHUBS_ID
az role assignment create --assignee $RECEIVER_SP_ID --role $EH_RECEIVER_ROLE_ID --scope $EVENTHUBS_ID

Alright, you're all set!

Let's try it out

Clone the sample apps from GitHub

git clone https://github.com/abhirockzz/azure-eventhubs-rbac-example.git

Start the Go consumer application:

export EVENTHUBS_NAMESPACE=[replace with namespace]
export EVENTHUB_NAME=[replace with name of the Event Hub]

export AZURE_TENANT_ID=[replace with Service Principal "tenant" for the receiver SP]
export AZURE_CLIENT_ID=[replace with Service Principal "appId" for the receiver SP]
export AZURE_CLIENT_SECRET=[replace with Service Principal "password" for the receiver SP]

cd azure-eventhubs-rbac-example/consumer-go

go run main.go

The program will block, waiting for events....

In another terminal, start producer application - this will just send 10 events end exit (re-run if you want to send more events)

cd azure-eventhubs-rbac-example/producer-java

//build the Java app - it uses Maven
mvn clean install

export EVENTHUBS_NAMESPACE=[replace with namespace]
export EVENTHUB_NAME=[replace with name of the Event Hub]

export AZURE_TENANT_ID=[replace with Service Principal "tenant" for the producer SP]
export AZURE_CLIENT_ID=[replace with Service Principal "appId" for the producer SP]
export AZURE_CLIENT_SECRET=[replace with Service Principal "password" for the producer SP]

java -jar target/eventhubs-java-producer-jar-with-dependencies.jar

You should see the received events in the consumer app terminal!

As an exercise, to simulate an error scenario, you exchange can the client details for sender and consumer apps to see how they behave.

Conclusion

Hopefully this was useful in demonstrating how to use RBAC for Event Hubs applications using Azure Active Directory. In a future blog post, I will try to cover Managed Identity as well. Until then, stay tuned!