You should use Docker image “confluentinc/kafka” 为了 Kafka 集群吗？

huangapple
go评论125阅读模式
英文:

why should i use docker image "confluentinc/kafka" to kafka cluster?

问题

我在本地使用以下命令进行Kafka集群的Docker容器化部署：

  1. docker-compose up

每当我想要创建主题、检索已有主题或搜索主题中存储的数据时，我使用以下命令：

  1. docker run --net=host --rm confluentinc/cp-kafka:latest kafka-topics --describe --topic bar --zookeeper localhost:32181

这些命令是在官方Kafka集群部署站点上使用的。然而，我真的想在本地存储中使用Kafka，而不是Kafka Docker镜像，如下所示：

  1. kafka-topics --describe --topic bar --zookeeper localhost:32181

但是，如果我使用上面的代码，我会遇到以下错误：

  1. Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING

此外，如果我检查netstat -anp tcp，我发现Kafka集群没有LISTEN端口。

我应该怎么做？请告诉我关于Docker的新手常见问题，因为我真的是Docker的新手 You should use Docker image “confluentinc/kafka” 为了 Kafka 集群吗？

以下是YAML配置文件：

  1. ---
  2. version: '2'
  3. services:
  4.   zookeeper-1:
  5.     image: confluentinc/cp-zookeeper:latest
  6.     environment:
  7.       ZOOKEEPER_SERVER_ID: 1
  8.       ZOOKEEPER_CLIENT_PORT: 22181
  9.       ZOOKEEPER_TICK_TIME: 2000
  10.       ZOOKEEPER_INIT_LIMIT: 5
  11.       ZOOKEEPER_SYNC_LIMIT: 2
  12.       ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
  13.     network_mode: host
  14.     extra_hosts:
  15.       - "moby:127.0.0.1"
  17.   zookeeper-2:
  18.     image: confluentinc/cp-zookeeper:latest
  19.     environment:
  20.       ZOOKEEPER_SERVER_ID: 2
  21.       ZOOKEEPER_CLIENT_PORT: 32181
  22.       ZOOKEEPER_TICK_TIME: 2000
  23.       ZOOKEEPER_INIT_LIMIT: 5
  24.       ZOOKEEPER_SYNC_LIMIT: 2
  25.       ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
  26.     network_mode: host
  27.     extra_hosts:
  28.       - "moby:127.0.0.1"
  30.   zookeeper-3:
  31.     image: confluentinc/cp-zookeeper:latest
  32.     environment:
  33.       ZOOKEEPER_SERVER_ID: 3
  34.       ZOOKEEPER_CLIENT_PORT: 42181
  35.       ZOOKEEPER_TICK_TIME: 2000
  36.       ZOOKEEPER_INIT_LIMIT: 5
  37.       ZOOKEEPER_SYNC_LIMIT: 2
  38.       ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
  39.     network_mode: host
  40.     extra_hosts:
  41.       - "moby:127.0.0.1"
  43.   kafka-1:
  44.     image: confluentinc/cp-kafka:latest
  45.     network_mode: host
  46.     depends_on:
  47.       - zookeeper-1
  48.       - zookeeper-2
  49.       - zookeeper-3
  50.     environment:
  51.       KAFKA_BROKER_ID: 1
  52.       KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
  53.       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092
  54.     extra_hosts:
  55.       - "moby:127.0.0.1"
  57.   kafka-2:
  58.     image: confluentinc/cp-kafka:latest
  59.     network_mode: host

希望这对你有所帮助，如果你有任何其他问题，请随时提问。

英文:

I did

  1. docker-compose up

for kafka clustering in my local.

and whenever i want to make topic, retrieve topics i have or search data stored in the topic i use

  1. docker run --net=host --rm confluentinc/cp-kafka:latest kafka-topics --describe --topic bar --zookeeper localhost:32181

which is on the official kafka clustering deployment site.

However i really want to use Kafka in my local storage not the kafka docker image like

  1. kafka-topics --describe --topic bar --zookeeper localhost:32181

if i use the code above i face this error.

  1. Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
  2.         at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)

Besides if i check netstat -anp tcp
there is no LISTEN ports from kafka cluster.

What am i supposed to do ?
and please let me know what i am missing now about docker (because i am really new to docker You should use Docker image “confluentinc/kafka” 为了 Kafka 集群吗？ )

This is yaml configuration

  1. ---
  2. version: '2'
  3. services:
  4.   zookeeper-1:
  5.     image: confluentinc/cp-zookeeper:latest
  6.     environment:
  7.       ZOOKEEPER_SERVER_ID: 1
  8.       ZOOKEEPER_CLIENT_PORT: 22181
  9.       ZOOKEEPER_TICK_TIME: 2000
  10.       ZOOKEEPER_INIT_LIMIT: 5
  11.       ZOOKEEPER_SYNC_LIMIT: 2
  12.       ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
  13.     network_mode: host
  14.     extra_hosts:
  15.       - "moby:127.0.0.1"
  17.   zookeeper-2:
  18.     image: confluentinc/cp-zookeeper:latest
  19.     environment:
  20.       ZOOKEEPER_SERVER_ID: 2
  21.       ZOOKEEPER_CLIENT_PORT: 32181
  22.       ZOOKEEPER_TICK_TIME: 2000
  23.       ZOOKEEPER_INIT_LIMIT: 5
  24.       ZOOKEEPER_SYNC_LIMIT: 2
  25.       ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
  26.     network_mode: host
  27.     extra_hosts:
  28.       - "moby:127.0.0.1"
  30.   zookeeper-3:
  31.     image: confluentinc/cp-zookeeper:latest
  32.     environment:
  33.       ZOOKEEPER_SERVER_ID: 3
  34.       ZOOKEEPER_CLIENT_PORT: 42181
  35.       ZOOKEEPER_TICK_TIME: 2000
  36.       ZOOKEEPER_INIT_LIMIT: 5
  37.       ZOOKEEPER_SYNC_LIMIT: 2
  38.       ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
  39.     network_mode: host
  40.     extra_hosts:
  41.       - "moby:127.0.0.1"
  43.   kafka-1:
  44.     image: confluentinc/cp-kafka:latest
  45.     network_mode: host
  46.     depends_on:
  47.       - zookeeper-1
  48.       - zookeeper-2
  49.       - zookeeper-3
  50.     environment:
  51.       KAFKA_BROKER_ID: 1
  52.       KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
  53.       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092
  54.     extra_hosts:
  55.       - "moby:127.0.0.1"
  57.   kafka-2:
  58.     image: confluentinc/cp-kafka:latest
  59.     network_mode: host
  60. "docker-compose.yml" 83L, 2321C

答案1

得分: 1

如果您正在主机上运行Kafka（和Zookeeper），那么不，您根本不需要Docker。

您不应在容器的任何连接字符串中使用 localhost，而应使用容器的外部服务名称，并从配置中删除 network_mode。

注意：在一台机器上运行多个代理/ Zookeeper 并没有什么帮助。

或者搜索存储在主题中的数据

您不应该使用Zookeeper来执行这项操作。

英文:

If you are running Kafka (and Zookeeper) on your host, then, no, you don't need Docker at all.

You should not use localhost in any connection string from the container, rather use the external service names of the containers and remove network_mode from the configurations.

Note: multiple brokers/zookeepers on one machine isn't helping anything.

> or search data stored in the topic

You shouldn't use zookeeper to do that

huangapple
  • 本文由 发表于 2020年1月6日 17:35:39
  • 转载请务必保留本文链接：https://go.coder-hub.com/59609677.html
  • apache-kafka
  • confluent-platform
  • docker
  • docker-compose
  • kafka-cluster
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定