Empowering Your Kafka Connectors: A Guide to Connector Guardian

Hi there :)

In this post, I want to introduce you to Connector Guardian. If you've ever found yourself grappling with the management of Kafka Connect connectors, you're in for a treat. Connector Guardian is tailor-made to simplify your life as a developer or operator, providing efficient tools for the seamless management and maintenance of your Kafka Connectors.

How It Works

Connector Guardian smoothly interacts with your Kafka Connect cluster using its REST API. In its initial release, version 0.1.0, it harnesses the power of jq for JSON parsing. Later, transitioning to version 0.2.0, Connector Guardian adopts Python's built-in JSON library.

Features

• Auto Connector Restart: Starting from V0.1.0, Connector Guardian monitors the status of connectors and tasks, automatically restarting them if they fail.

• Restart Back Off: Introduced in V0.3.0, this feature ensures that restarts occur at increasing time intervals. The initial restart happens immediately, and subsequent restarts are delayed exponentially. This approach allows for efficient issue resolution, even in the face of prolonged network outages. After a configurable number of restarts (MAX_RESTART), the Guardian stops automatic restarting, leaving it to you for manual intervention.

How to Add Guardian to Kafka Connect Cluster

Container Image

You can easily pull the Connector Guardian image from Docker Hub, then run it with docker run command:

docker run -d \
  -e KAFKA_CONNECT_HOST=localhost \
  -e KAFKA_CONNECT_PORT=8083 \
  -e KAFKA_CONNECT_PROTO=http \
  -e KAFKA_CONNECT_USER='' \
  -e KAFKA_CONNECT_PASS='' \
  -e ENABLE_BACKOFF=1 \
  -e MAX_RESTART=7 \
  -e EXPONENTIAL_RATIO=2 \
  anvaari/connector-guardian

Non-Cloud Environments

For deployment on your server, use the provided docker-compose file. Before deploying the image, ensure that you set the appropriate environment variables in docker-compose.yaml.

cd deploy
docker compose up -d

Kubernetes or OpenShift

Utilize the provided Helm chart for deployment. Make sure to set the required environment variables in values.yaml.

helm upgrade connector-guardian --install -n {your_namespace_name} -f deploy/chart/values.yaml deploy/chart

Once deployed, Connector Guardian runs as a pod, executing connector_guardian.py every 5 minutes.

Environment Variables

To use the Docker image, docker-compose, or Helm chart, set the following environment variables:

• KAFKA_CONNECT_HOST: Default = localhost
• KAFKA_CONNECT_PORT: Default = 8083
• KAFKA_CONNECT_PROTO: Default = http
• KAFKA_CONNECT_USER: Default = ''
• KAFKA_CONNECT_PASS: Default = ''
• ENABLE_BACKOFF: Default = 1
• MAX_RESTART: Default = 7
• EXPONENTIAL_RATIO: Default = 2

In The End ...

Connector Guardian is your steadfast companion in the realm of Kafka Connect connectors. Whether you are a seasoned developer or an operations expert, this tool streamlines the management of your connectors, offering an automated approach to restarts and intelligent back-off mechanisms.

As I continue to evolve Connector Guardian, I invite you to be part of this journey. Your feedback, suggestions, and contributions are not only valued but crucial in shaping the future of this open-source project. Let's work together to make Kafka Connector maintenance a seamless experience for all.

Get Involved!

• Contribute to the project on GitHub.
• Share your experiences and ideas in the issues section.
• Spread the word - let others in your network know about Connector Guardian.
• Stay tuned for updates and new features!