Step-by-Step: Deploy the most advanced Kafka Console on AWS in minutes

James White — Fri, 23 Feb 2024 12:45:44 +0000

Launching Conduktor in minutes using CloudFormation and ECS Fargate.

Overview

Utilizing a console for managing your Kafka clusters, checking topic configs, monitoring lag, and analyzing your Kafka data can save you a lot of time. You get a complete picture through one pane of glass, as opposed to wrangling with a multitude of CLIs, APIs and devtools.

Conduktor is the most advanced console for getting insight into your Kafka ecosystem. It’s a vendor agnostic solution, meaning it works with Confluent, AWS MSK, Redpanda and all other Kafka protocol compliant providers. It also supports Confluent Schema Registry and AWS Glue, as well as Kafka Connect and ksqlDB.

This guide will show you how to get a free version of Conduktor up and running alongside all dependencies on ECS in minutes. Using CloudFormation, the process is dead simple and requires only a few clicks. If you prefer Kubernetes, try our Helm chart.

Networking & Architecture

The CloudFormation script will deploy an RDS instance for storing metadata such as user preferences, resource tags, users, groups and permissions.

The Conduktor service will be exposed on a public IP address by default, though you can edit this if it's not suitable for your need.

Deployment Steps

While this guide will help you get started, you may need to make additional configurations to ensure your deployment is production-ready.

To start the deployment, please go to https://aws.amazon.com/cloudformation/getting-started/
Copy/sync from github using one of the following CloudFormation templates:
- Full - use this to deploy Conduktor and to spin up the infrastructure needed to run it.
- Lite - use this to deploy Conduktor on existing infrastructure (note you will need an ECS Cluster and an RDS database).

Wait for the stack to be created. This should take 5-10 minutes.
Go to Cluster → Service → Task → Conduktor Console → Network Bindings and click on the address.

You can now onboard onto Console. The first step is to add your Kafka cluster configuration.

In the cluster configuration screen, add your Kafka clusters bootstrap server and authentication details.
- If you are using Schema registry, Kafka Connect or ksqlDB, you should also add these details via the tabs highlighted below.

Final step! Invite your team mates to collaborate with you from the Users & Groups screen.
- Note you can configure SSO (LDAP, OIDC) or add users via basic authentication.

That's a wrap!

You're all ready to go with the most powerful Kafka Console on the market. Start drilling down into your Kafka data, managing your subjects, connectors, consumer groups, service accounts and more!

Refer to the documentation for more detailed steps and other deployment options such as Docker and Helm.

DEV Community: James White

Step-by-Step: Deploy the most advanced Kafka Console on AWS in minutes

Overview

Networking & Architecture

Deployment Steps

That's a wrap!