We try to Integrate our existing Kafka server to our SSIS services, due we do complex transformation process to our Kafka messages which cannot meet our expectation for low latency when we use manual Python programming.
The job is to trigger SSIS process when there is new messages in Kafka server. Our message is in JSON format due the Kafka source is Debezium CDC source from our PostgreSQL. SSIS will breakdown the message into several new row and insert it into representative destination tables. As there's no Kafka Connectors to SSIS, so we use scripting task.
We use Confluent Client as our backbone for scripting the connectors, as it was also available with Apache Kafka, but we confused how to push the messages directly into SSIS, so we can make our Kafka server as a data source to our SSIS with this script and trigger SSIS process if there's new message come from Kafka. We still new using scripting for Kafka connection in SSIS due lack of example sources.
Top comments (0)