Kafka Connect is becoming a force on the Change Data Capture field.
Kafka itself has gained a lot of momentum being more and more adopted by Com...
For further actions, you may consider blocking this person and/or reporting abuse
do you have any idea why my dockerized asp.net core app can't connect to kafka's localhost:9092, but if not dockerized it can successfully connect?
e.g.
docker run -p 5001:80 --rm my-api
my bootstrap server is: BootstrapServers = "127.0.0.1:9092"
Hi, most likely related to the KAFKA_ADVERTISED_LISTENERS when using a docker container which enables you to pass in the advertised.listeners values to kafka, notice that this is a comma-separated list of listeners with their the host/ip and port and it's metadata that’s passed back to clients, so check your docker image settings, which docker image are you using?
kafka.apache.org/documentation/#br...
Also make sure to be exposing the internal kafka docker port.
Sorry for the late reply, got this working already Thanks!
I`ve followed that instraction, but have got the error:
PS D:\9_soft\docker\streaming_kafka_with_docker\win_kafka_manager\kafka_connect_docker> docker-compose build
zookeeper uses an image, skipping
kafka uses an image, skipping
Building connect-standalone
Step 1/5 : FROM wurstmeister/kafka:2.12-2.3.0
---> 48591fe02a72
Step 2/5 : COPY connect-standalone.properties /opt/kafka/config/
---> Using cache
---> 5c91bc255d31
Step 3/5 : COPY connect-file-sink.properties /opt/kafka/config/
---> Using cache
---> 32044e2edb55
Step 4/5 : COPY start-kafka.sh /opt/kafka/bin/
---> Using cache
---> 6f1c44233efc
Step 5/5 : RUN chmod a+x /usr/bin/start-kafka.sh
---> Using cache
---> 45a188c5f8bc
Successfully built 45a188c5f8bc
Successfully tagged kafka_connect_docker_connect-standalone:latest
PS D:\9_soft\docker\streaming_kafka_with_docker\win_kafka_manager\kafka_connect_docker> docker-compose up -d
Creating network "kafka_connect_docker_default" with the default driver
Creating zookeeper ... done Creating kafka ... done Creating connect-standalone ... done PS D:\9_soft\docker\streaming_kafka_with_docker\win_kafka_manager\kafka_connect_docker> docker-compose ps
Name Command State Ports
connect-standalone start-kafka.sh Exit 1
kafka start-kafka.sh Up 0.0.0.0:9092->9092/tcp
zookeeper /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp
PS D:\9_soft\docker\streaming_kafka_with_docker\win_kafka_manager\kafka_connect_docker> docker-compose logs connect-standalone
Attaching to connect-standalone
connect-standalone | ERROR: missing mandatory config: KAFKA_ZOOKEEPER_CONNECT
Hum.. That's unfortunate, I will run this again and double check but definitely worked before? What O.S are you running? Did you check the ports if they're all available? Did you update any versions of the images in the compose-file before running? Because KAFKA_ZOOKEEPER_CONNECT is actually defined.
Also I would check your yml file for proper identation and formatting, yml can be very tricky for those. Make sure it's compliant. I've never used but you could try this one: yamllint.com/ or add a yml lint to your IDE and make sure it's properly formatted.
If will set bootstrap.servers=kafka:9092 in my connect-standalone.properties file and copy it to the connect container how is going to resolve the kafka broker ip ? I am getting following error as it is not able to resolve.
WARN Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka (org.apache.kafka.clients.ClientUtils:74)
If you have problems to connect your kafka client from your local development machine to the broker running in docker or Kubernetes please check the section
Connecting a Kafka clienton the end of this other post where you'll find details on how to fix it and links explaining why it happens and sort this out completely, have fun.Hi Marcos Maia,
Loved your post. I have recently started working with Kafka and docker. Your post was a saviour for me. Post is very well written.
I just want to know is it possible with the kafka image I have to write to a http api as sink. I want a sink connector and process with which we write the data from the topic to an api. Could you help me out with that?
Hi,
Well, you could, Confluent even has a connector for that. docs.confluent.io/5.0.4/connect/ka... but unfortunatelly it's a paid connector.
Usually for this use cases where you want to get data from kafka and send using HTTP I would most likely go for a traditional consumer where you have more control over the data you can also control offset commits back to kafka if you need delivery garantees (i.e that the endpoint calls 200 on the call) which would be simpler with a standard kafka consumer.
I have articles on kafka consumers I would recommend you to take a look at them, also you will find tutorials on building http clients which you could connect to your consumer to achieve what you want.
Best,