How To Run Spring Cloud Data Flow with Azure Event Hub as a Binder

mimiz33 — Sun, 09 Aug 2020 14:19:24 +0000

Recently I had to work with Spring Cloud Data Flow, which is a microservice streaming and batch platform based on Cloud Foundry and Kubernetes. As I am using it with Kubernetes, I will not talk about Cloud Foundry.

I am using Azure Kubernetes Service, and I would like to use Azure Event Hubs as Spring Cloud Data Flow Binder.

We (me and my teammates) spent a lot of time reading documentation (Spring Cloud Data Flow configuration and also Spring Cloud Stream configurations ) and we've found that we should define binders configuration when deploying the stream, and not at a server level.

Imagine the following stream:

stream create --name 'uppercase' --definition 'http | transform --expression=payload.toUpperCase() | log'

This stream will get some data from an http endpoint, then the data will be transformed to uppercase then logged, an easy stream, good for make some tests.

Note: Each application (http/transform/log) need to have a dependency to the specific binder (and only one), it means that if you want to use the same application with different binders you will have to build different versions.

When deploying the stream, you can pass some configuration properties to each application :

stream deploy uppercase --properties "deployer.http.kubernetes.createNodePort=true"

This for example will tell the "deployer" (ie: Skipper) to create a NodePort service pointing to the http application in Kubernetes (so it will be possible to send a message from outside the cluster)

We want that when the application is created by Spring Cloud Data Flow, it passes some properties directly to the applications (and all applications), for example, we do not want to define the credential information for each application when deploying the stream, it may be injected by Spring Cloud Data Flow itself.

In order to do that we will have to edit the ConfigMap used for the Dataflow Server, to define a set of binders (with all needed configuration), and then it will be possible to reference those binders when deploying our stream.

Here is the new ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: scdf-server
  labels:
    app: scdf-server
data:
  application.yaml: |-
    spring:
      cloud:
        dataflow:
          applicationProperties:
            stream:
              spring:
                cloud:
                  azure:
                    eventhub:
                      # this is the "default" binders as it is needed by
                      # The Azure Event Hub Binder
                      connection-string: BINDER_ENDPOINT_CONNECTION_STRING
                      checkpoint-storage-account: BINDER_STORAGE_ACCOUNT
                      checkpoint-access-key: BINDER_STORAGE_ACCOUNT_ACCESS_KEY
                      checkpoint-container: BINDER_STORAGE_ACCOUNT_CONTAINER
                  stream:
                    # create other binders if needed
                    binders:
                      binder-one:
                        type: eventhub
                        defaultCandidate: false
                        environment:
                          spring:
                            cloud:
                              azure:
                                eventhub:
                                  connection-string: BINDER_ONE_ENDPOINT_CONNECTION_STRING
                                  checkpoint-storage-account: BINDER_ONE_STORAGE_ACCOUNT
                                  checkpoint-access-key: BINDER_ONE_STORAGE_ACCOUNT_ACCESS_KEY
                                  checkpoint-container: BINDER_ONE_STORAGE_ACCOUNT_CONTAINER
                       binder-two:
                         type: eventhub
                         defaultCandidate: false
                         environment:
                           spring:
                             cloud:
                               azure:
                                 eventhub:
                                   connection-string: BINDER_TWO_ENDPOINT_CONNECTION_STRING
                                   checkpoint-storage-account: BINDER_TWO_STORAGE_ACCOUNT
                                   checkpoint-access-key: BINDER_TWO_STORAGE_ACCOUNT_ACCESS_KEY
                                   checkpoint-container: BINDER_TWO_STORAGE_ACCOUNT_CONTAINER
      datasource:
        url: jdbc:sqlserver:
        username: your_username
        password: your_password
        driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
        testOnBorrow: true
        validationQuery: "SELECT 1"

Then now when deploying the stream you can use the following command :

stream deploy uppercase --properties "deployer.http.kubernetes.createNodePort=true,app.http.spring.cloud.stream.bindings.output.binder=binder-one,app.transform.spring.cloud.stream.bindings.input.binder=binder-one,app.transform.spring.cloud.stream.bindings.output.binder=binder-two,app.log.spring.cloud.stream.bindings.input.binder=binder-two"

This is it!

I hope this post helped you.
Note, I originally posted this article on medium : https://medium.com/@rgoyard/how-to-run-spring-cloud-data-flow-with-azure-event-hub-as-a-binder-f5e888d0719

DEV Community: mimiz33

How To Run Spring Cloud Data Flow with Azure Event Hub as a Binder