How to start a dockerized Kafka instance and connect to it from the host machine?

How do I do the following:

• Start a dockerized instance of Kafka on host machine1
• Run a process on machine1 which can consume or produce Kafka messages

Here is what I have so far:

• I installed the following items on my host machine. (machine1):
• docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
• This is a Linux computer running Debian 11. It exists on my local network. I am connected to it from a Windows computer via ssh.
• I added my user to the group docker.
• I downloaded the following YAML file to obtain a docker-compose.yml file:
• curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/kafka/docker-compose.yml > docker-compose.yml
• That looks like this:

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3.4
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

I don't know much about Docker at all, and my experience with it is limited, however this looks ok. I would like to use the bitnami containers, as I have worked with these before, albeit in a limited capacity.

I note that the port numbers look correct.

• Finally, I started the docker containers with docker compose up -d
• I can see the containers started with docker ps

$ docker ps -a
CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                                     NAMES
b56084cdd443   bitnami/kafka:3.4       "/opt/bitnami/script…"   3 seconds ago   Up 2 seconds   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                 kafka-kafka-1
0e6b30a47d06   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   4 seconds ago   Up 3 seconds   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp   kafka-zookeeper-1

What works:

I found that I was able to create a Kafka Topic by executing a bash session inside of the Kafka container. I was also able to Produce and Consume messages using the helper utilities.

The three utilities of interest live here:

$ docker exec -it kafka-kafka-1 bash
I have no name!@b56084cdd443:$ cd /opt/bitnami/kafka/bin
I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ls
...
kafka-console-consumer.sh
kafka-console-producer.sh
kafka-topics.sh

They appear to be working. For example, I created test-topic with this command:

I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ./kafka-topics.sh 
 \
--bootstrap-server localhost:9092 --create --topic test-topic

What does not work:

• I was not able to Consume or Produce messages using kafkacat

This was my next stage in testing.

• I installed kafkacat on the host machine (machine1):
• sudo apt install kafkacat
• I ran kafkacat to inspect the topics: kafkacat -L -t test-topic -p localhost:9092

$ kafkacat -L -t test-topic -b localhost:9092
Metadata for test-topic (from broker -1: localhost:9092/bootstrap):
 1 brokers:
  broker 1001 at b56084cdd443:9092 (controller)
 1 topics:
  topic "test-topic" with 1 partitions:
    partition 0, leader 1001, replicas: 1001, isrs: 1001

That seems to be ok.

I then became stuck trying to Produce messages and send them to Kafka.

$ kafkacat -P -b localhost:9092 -t test-topic
Hello World
%3|1680812310.642|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: 
b56084cdd443:9092/1001: Failed to resolve 'b56084cdd443:9092': 
Name or service not known (after 36ms in state CONNECT)
%3|1680812311.658|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: b56084cdd443:9092/1001: 
Failed to resolve 'b56084cdd443:9092': 
Name or service not known (after 15ms in state CONNECT, 1 identical error(s) suppressed)
^\Quit

At this point I reach a blocker and don't know how to proceed.

While trying to research the problem, I found an article which initially looked useful, but wasn't. I didn't understand it, probably because I don't have sufficient background information.

https://www.baeldung.com/kafka-docker-connection

If this article indeed does explain what is wrong, and how to fix it, then could someone explain it to me in further detail so that I can understand what to do and why this doesn't work in its current configuration?

If it turns out not to be relevant then please ignore it.

I found a solution, which required some editing of my docker-compose.yml file.

$ cat docker-compose.yml
version: "2"

services:
  zookeeper:
    container_name: zookeeper1
    networks:
      - "kafka_net"
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    container_name: kafka1
    networks:
      - "kafka_net"
    image: docker.io/bitnami/kafka:3.4
    ports:
      #- "9092:9092"
      - 29092:29092
    volumes:
      - "kafka_data:/bitnami"
    environment:
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
      ALLOW_PLAINTEXT_LISTENER: yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

networks:
  kafka_net:
    name: kafka_net

We can see this now works as expected:

kafkacat -L -t test-topic -b localhost:29092
Metadata for test-topic (from broker 1001: localhost:29092/1001):
 1 brokers:
  broker 1001 at localhost:29092 (controller)
 1 topics:
  topic "test-topic" with 1 partitions:
    partition 0, leader 1001, replicas: 1001, isrs: 1001

And more importantly...

$ kafkacat -P -b localhost:29092 -t test-topic
hello world
^\Quit

$ kafkacat -C -b localhost:29092 -t test-topic
Hello World
These are two test messages.
hello world
hello world
% Reached end of topic test-topic [0] at offset 4

Note that "hello world" is shown twice. One entry was from testing the dockerized kafka-console-producer.sh, the second is from kafkacat.