Subject filtering on blob events in Azure Event Grid

We can implement simple, but powerful event routing in Azure Event Grid thanks to Subject filtering

I’m currently working on a personal project that reads various different csv files from a local directory, uploads the files to Azure Blob Storage and then persists the records of each file into Azure Cosmos DB. These are downloaded files from my Fitbit dashboard.

At a high level, The architecture looks like this:

The idea here is that I have .NET Core Docker applications that watch for new files in a local directory and then upload that file to Blob Storage. Once the blob is created, this would trigger an event that Event Grid sends to an Azure Function, which then reads each line in the CSV file and persists that line as a record into Azure Cosmos DB.

I’ve worked with Event Grid before, so I knew it was pretty straightforward to setup a topic on my Azure Storage account and then create a subscription for my Azure Functions that would handle the incoming events. Azure Functions can be invoked using an Event Grid trigger, which makes it super easy to setup and work with events from Event Grid.

For this personal project, what I needed to do what trigger different types of events depending on what file I had uploaded to Blob Storage. The last thing I wanted to do was set up multiple file watchers that handle a specific type of file to it’s own storage account and then set up a Event Grid Topic for each storage account.

Instead of wasting our time doing all that, we can use subject filtering to send different events depending on what blob was uploaded to Azure Storage.

In my storage account, I have a container where all my files are being uploaded to. Within this container, I will have one folder for all the sleep files and another for all the activity files.

So what I needed to do here is filter on a blob prefix (name of the directory in blob storage) and send an event every time a blob is created within that directory. This is possible by filtering the event by the subject that within the event data.

This subject can be found within the Event Grid schema. The schema does change slightly depending on the event that was emitted, but the schema generally looks like this:

[
  {
    "topic": string,
    "subject": string,
    "id": string,
    "eventType": string,
    "eventTime": string,
    "data":{
      object-unique-to-each-publisher
    },
    "dataVersion": string,
    "metadataVersion": string
  }
]

When we filter an event by the subject, we can specify a value to match at the beginning of the subject or the end of the subject.

Applying subject filtering on Event Grid is simple to set up in the portal. I’ve gone ahead and set up an Event Grid System Topic, so all we need to do is create new event subscribers that use subject filtering. Go into your Event Grid Topic and click “Event Subscription” to get started.

Give your event subscription a name and use the Event Grid Schema as the event schema. My Event Grid System Topic is applied on my storage account and I only want events to be triggered when a blob is created. I’ve also picked my ProcessMonthlyActivityFile function as my endpoint.

Once you’ve set up the basics, click on the Filters tab. Click on the Enable Subject Filtering checkbox and you’ll be able to configure subject filtering.

For my application, I want to be able to filter on a blob prefix, which takes the form of /blobServices/default/containers/storageaccountname/blobs/blobprefix.

If we wanted to filter on just the container, we would just use /blobServices/default/containers/storageaccountname/. Subscribers to your event use the subject property within the Event Grid schema to determine how and which event should be routed to its configured endpoint.

Click on Create to finish creating your filter. You’ll see them created within your Event Grid system topic.

I’ve set up two filters within my Event Grid System Topic so that when activity files are created within my Storage Account, they are routed to the ProcessMonthlyActivityFile function. When sleep files are created, they will be routed to the ProcessSleepFile function.

We can verify that this has taken place by checking the Function’s logs. Here’s the logs from our ProcessMonthlyActivityFile function:

And here are the logs for our ProcessSleepFile function:

Within the functions themselves, there is no special code to filter events for us. All of the filtering is done within Event Grid, so this is a really powerful and simple way of implementing routing of events without having to write your own clients to do so! (Pretty cool 😎)

You can also filter events in Event Grid by Event type or by using more advanced methods. If you want to learn how to do that, you should read this article.

If you want to learn more about Azure Event Grid and how to work with it, I’d recommend you check out the following articles:

• What is Azure Event Grid?
• Quickstart: Route custom events to an Azure Function with Event Grid

If you have any questions, please feel free to comment below or reach out to me on Twitter.