如何启动一个Docker化的Kafka实例并从主机机器连接到它？

huangapple
go评论88阅读模式
英文:

How to start a dockerized Kafka instance and connect to it from the host machine?

问题

以下是翻译的部分：

如何执行以下操作：

  • 在主机machine1上启动一个Docker化的Kafka实例
  • 在machine1上运行一个进程，可以消费或生产Kafka消息

到目前为止，我做了以下事情：

  • 我在我的主机（machine1）上安装了以下项目：
    • docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  • 这是一台运行Debian 11的Linux计算机，它存在于我的本地网络中。我通过SSH从Windows计算机连接到它。
  • 我将我的用户添加到docker组。
  • 我下载了以下YAML文件以获取docker-compose.yml文件：
    • curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/kafka/docker-compose.yml > docker-compose.yml
  • 它看起来像这样：
  1. version: "2"
  3. services:
  4.   zookeeper:
  5.     image: docker.io/bitnami/zookeeper:3.8
  6.     ports:
  7.       - "2181:2181"
  8.     volumes:
  9.       - "zookeeper_data:/bitnami"
  10.     environment:
  11.       - ALLOW_ANONYMOUS_LOGIN=yes
  12.   kafka:
  13.     image: docker.io/bitnami/kafka:3.4
  14.     ports:
  15.       - "9092:9092"
  16.     volumes:
  17.       - "kafka_data:/bitnami"
  18.     environment:
  19.       - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
  20.       - ALLOW_PLAINTEXT_LISTENER=yes
  21.     depends_on:
  22.       - zookeeper
  24. volumes:
  25.   zookeeper_data:
  26.     driver: local
  27.   kafka_data:
  28.     driver: local

我对Docker知之甚少，我的经验有限，但这看起来还可以。我想使用bitnami容器，因为我以前有过有限的使用经验。

我注意到端口号看起来是正确的。

  • 最后，我使用docker compose up -d启动了Docker容器
  • 我可以用docker ps看到容器已经启动
  1. $ docker ps -a
  2. CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                                     NAMES
  3. b56084cdd443   bitnami/kafka:3.4       "/opt/bitnami/script…"   3 seconds ago   Up 2 seconds   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                 kafka-kafka-1
  4. 0e6b30a47d06   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   4 seconds ago   Up 3 seconds   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp   kafka-zookeeper-1

什么有效：

我发现我能够通过在Kafka容器内执行bash会话来创建Kafka主题。我还能够使用辅助工具生产和消费消息。

感兴趣的三个实用程序位于此处：

  1. $ docker exec -it kafka-kafka-1 bash
  2. I have no name!@b56084cdd443:$ cd /opt/bitnami/kafka/bin
  3. I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ls
  4. ...
  5. kafka-console-consumer.sh
  6. kafka-console-producer.sh
  7. kafka-topics.sh

它们似乎正在工作。例如，我使用以下命令创建了test-topic：

  1. I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ./kafka-topics.sh 
  2.  \
  3. --bootstrap-server localhost:9092 --create --topic test-topic

什么不起作用：

  • 我无法使用kafkacat来生产或消费消息

这是我测试的下一个阶段。

  • 我在主机（machine1）上安装了kafkacat：
  • sudo apt install kafkacat
  • 我运行kafkacat来检查主题：kafkacat -L -t test-topic -p localhost:9092
  1. $ kafkacat -L -t test-topic -b localhost:9092
  2. Metadata for test-topic (from broker -1: localhost:9092/bootstrap):
  3.  1 brokers:
  4.   broker 1001 at b56084cdd443:9092 (controller)
  5.  1 topics:
  6.   topic "test-topic" with 1 partitions:
  7.     partition 0, leader 1001, replicas: 1001, isrs: 1001

这看起来还可以。

然后，我陷入困境，试图生产消息并将其发送到Kafka。

  1. $ kafkacat -P -b localhost:9092 -t test-topic
  2. Hello World
  3. %3|1680812310.642|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: 
  4. b56084cdd443:9092/1001: Failed to resolve 'b56084cdd443:9092': 
  5. Name or service not known (after 36ms in state CONNECT)
  6. %3|1680812311.658|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: b56084cdd443:9092/1001: 
  7. Failed to resolve 'b56084cdd443:9092': 
  8. Name or service not known (after 15ms in state CONNECT, 1 identical error(s) suppressed)
  9. ^\Quit

到此为止，我遇到了障碍，不知道如何继续。

英文:

How do I do the following:

  • Start a dockerized instance of Kafka on host machine1
  • Run a process on machine1 which can consume or produce Kafka messages

Here is what I have so far:

  • I installed the following items on my host machine. (machine1):
  • docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
  • This is a Linux computer running Debian 11. It exists on my local network. I am connected to it from a Windows computer via ssh.
  • I added my user to the group docker.
  • I downloaded the following YAML file to obtain a docker-compose.yml file:
  • curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/kafka/docker-compose.yml > docker-compose.yml
  • That looks like this:
  1. version: "2"
  3. services:
  4.   zookeeper:
  5.     image: docker.io/bitnami/zookeeper:3.8
  6.     ports:
  7.       - "2181:2181"
  8.     volumes:
  9.       - "zookeeper_data:/bitnami"
  10.     environment:
  11.       - ALLOW_ANONYMOUS_LOGIN=yes
  12.   kafka:
  13.     image: docker.io/bitnami/kafka:3.4
  14.     ports:
  15.       - "9092:9092"
  16.     volumes:
  17.       - "kafka_data:/bitnami"
  18.     environment:
  19.       - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
  20.       - ALLOW_PLAINTEXT_LISTENER=yes
  21.     depends_on:
  22.       - zookeeper
  24. volumes:
  25.   zookeeper_data:
  26.     driver: local
  27.   kafka_data:
  28.     driver: local

I don't know much about Docker at all, and my experience with it is limited, however this looks ok. I would like to use the bitnami containers, as I have worked with these before, albeit in a limited capacity.

I note that the port numbers look correct.

  • Finally, I started the docker containers with docker compose up -d
  • I can see the containers started with docker ps
  1. $ docker ps -a
  2. CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                                     NAMES
  3. b56084cdd443   bitnami/kafka:3.4       "/opt/bitnami/script…"   3 seconds ago   Up 2 seconds   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                 kafka-kafka-1
  4. 0e6b30a47d06   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   4 seconds ago   Up 3 seconds   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp   kafka-zookeeper-1

What works:

I found that I was able to create a Kafka Topic by executing a bash session inside of the Kafka container. I was also able to Produce and Consume messages using the helper utilities.

The three utilities of interest live here:

  1. $ docker exec -it kafka-kafka-1 bash
  2. I have no name!@b56084cdd443:$ cd /opt/bitnami/kafka/bin
  3. I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ls
  4. ...
  5. kafka-console-consumer.sh
  6. kafka-console-producer.sh
  7. kafka-topics.sh

They appear to be working. For example, I created test-topic with this command:

  1. I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ./kafka-topics.sh 
  2.  \
  3. --bootstrap-server localhost:9092 --create --topic test-topic

What does not work:

  • I was not able to Consume or Produce messages using kafkacat

This was my next stage in testing.

  • I installed kafkacat on the host machine (machine1):
  • sudo apt install kafkacat
  • I ran kafkacat to inspect the topics: kafkacat -L -t test-topic -p localhost:9092
  1. $ kafkacat -L -t test-topic -b localhost:9092
  2. Metadata for test-topic (from broker -1: localhost:9092/bootstrap):
  3.  1 brokers:
  4.   broker 1001 at b56084cdd443:9092 (controller)
  5.  1 topics:
  6.   topic "test-topic" with 1 partitions:
  7.     partition 0, leader 1001, replicas: 1001, isrs: 1001

That seems to be ok.

I then became stuck trying to Produce messages and send them to Kafka.

  1. $ kafkacat -P -b localhost:9092 -t test-topic
  2. Hello World
  3. %3|1680812310.642|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: 
  4. b56084cdd443:9092/1001: Failed to resolve 'b56084cdd443:9092': 
  5. Name or service not known (after 36ms in state CONNECT)
  6. %3|1680812311.658|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: b56084cdd443:9092/1001: 
  7. Failed to resolve 'b56084cdd443:9092': 
  8. Name or service not known (after 15ms in state CONNECT, 1 identical error(s) suppressed)
  9. ^\Quit

At this point I reach a blocker and don't know how to proceed.

While trying to research the problem, I found an article which initially looked useful, but wasn't. I didn't understand it, probably because I don't have sufficient background information.

https://www.baeldung.com/kafka-docker-connection

If this article indeed does explain what is wrong, and how to fix it, then could someone explain it to me in further detail so that I can understand what to do and why this doesn't work in its current configuration?

If it turns out not to be relevant then please ignore it.

答案1

得分: 0

I found a solution, which required some editing of my docker-compose.yml file.

  1. $ cat docker-compose.yml
  2. version: "2"
  4. services:
  5.   zookeeper:
  6.     container_name: zookeeper1
  7.     networks:
  8.       - "kafka_net"
  9.     image: docker.io/bitnami/zookeeper:3.8
  10.     ports:
  11.       - "2181:2181"
  12.     volumes:
  13.       - "zookeeper_data:/bitnami"
  14.     environment:
  15.       - ALLOW_ANONYMOUS_LOGIN=yes
  16.   kafka:
  17.     container_name: kafka1
  18.     networks:
  19.       - "kafka_net"
  20.     image: docker.io/bitnami/kafka:3.4
  21.     ports:
  22.       #- "9092:9092"
  23.       - 29092:29092
  24.     volumes:
  25.       - "kafka_data:/bitnami"
  26.     environment:
  27.       KAFKA_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
  28.       KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092
  29.       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
  30.       KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  31.       KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
  32.       ALLOW_PLAINTEXT_LISTENER: yes
  33.     depends_on:
  34.       - zookeeper
  36. volumes:
  37.   zookeeper_data:
  38.     driver: local
  39.   kafka_data:
  40.     driver: local
  42. networks:
  43.   kafka_net:
  44.     name: kafka_net

We can see this now works as expected:

  1. kafkacat -L -t test-topic -b localhost:29092
  2. Metadata for test-topic (from broker 1001: localhost:29092/1001):
  3.  1 brokers:
  4.   broker 1001 at localhost:29092 (controller)
  5.  1 topics:
  6.   topic "test-topic" with 1 partitions:
  7.     partition 0, leader 1001, replicas: 1001, isrs: 1001

And more importantly...

  1. $ kafkacat -P -b localhost:29092 -t test-topic
  2. hello world
  3. ^\Quit
  1. $ kafkacat -C -b localhost:29092 -t test-topic
  2. Hello World
  3. These are two test messages.
  4. hello world
  5. hello world
  6. % Reached end of topic test-topic [0] at offset 4

Note that "hello world" is shown twice. One entry was from testing the dockerized kafka-console-producer.sh, the second is from kafkacat.

英文:

I found a solution, which required some editing of my docker-compose.yml file.

  1. $ cat docker-compose.yml
  2. version: "2"
  4. services:
  5.   zookeeper:
  6.     container_name: zookeeper1
  7.     networks:
  8.       - "kafka_net"
  9.     image: docker.io/bitnami/zookeeper:3.8
  10.     ports:
  11.       - "2181:2181"
  12.     volumes:
  13.       - "zookeeper_data:/bitnami"
  14.     environment:
  15.       - ALLOW_ANONYMOUS_LOGIN=yes
  16.   kafka:
  17.     container_name: kafka1
  18.     networks:
  19.       - "kafka_net"
  20.     image: docker.io/bitnami/kafka:3.4
  21.     ports:
  22.       #- "9092:9092"
  23.       - 29092:29092
  24.     volumes:
  25.       - "kafka_data:/bitnami"
  26.     environment:
  27.       KAFKA_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
  28.       KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092
  29.       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
  30.       KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  31.       KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
  32.       ALLOW_PLAINTEXT_LISTENER: yes
  33.     depends_on:
  34.       - zookeeper
  36. volumes:
  37.   zookeeper_data:
  38.     driver: local
  39.   kafka_data:
  40.     driver: local
  42. networks:
  43.   kafka_net:
  44.     name: kafka_net

We can see this now works as expected:

  1. kafkacat -L -t test-topic -b localhost:29092
  2. Metadata for test-topic (from broker 1001: localhost:29092/1001):
  3.  1 brokers:
  4.   broker 1001 at localhost:29092 (controller)
  5.  1 topics:
  6.   topic "test-topic" with 1 partitions:
  7.     partition 0, leader 1001, replicas: 1001, isrs: 1001

And more importantly...

  1. $ kafkacat -P -b localhost:29092 -t test-topic
  2. hello world
  3. ^\Quit
  1. $ kafkacat -C -b localhost:29092 -t test-topic
  2. Hello World
  3. These are two test messages.
  4. hello world
  5. hello world
  6. % Reached end of topic test-topic [0] at offset 4

Note that "hello world" is shown twice. One entry was from testing the dockerized kafka-console-producer.sh, the second is from kafkacat.

huangapple
  • 本文由 发表于 2023年4月7日 04:20:56
  • 转载请务必保留本文链接：https://go.coder-hub.com/75953459.html
  • apache-kafka
  • docker
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定