为什么外部连接到Kafka不起作用？

huangapple
go评论168阅读模式
英文:

Why doesn't work external connection to Kafka?

问题

我在K8s集群中创建了Kafka服务和Kafka Pod，并部署了我的应用程序。因此，我可以从同一K8s集群中的应用程序读/写Kafka消息，但无法使用Kafka-tool或offset-explorer等外部工具连接到Kafka。以下是我的Kafka Pod和Service的YAML配置：

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   labels:
  5.     app: kafka-service
  6.   name: kafka-service
  7. spec:
  8.   type: NodePort
  9.   selector:
  10.     app: kafka-broker
  11.   ports:
  12.     - name: kafka-port
  13.       port: 9092
  14.       targetPort: 9092
  15.       nodePort: 30126
  16. ---
  17. apiVersion: apps/v1
  18. kind: Deployment
  19. metadata:
  20.   labels:
  21.     app: kafka-broker
  22.   name: kafka-broker
  23. spec:
  24.   replicas: 1
  25.   selector:
  26.     matchLabels:
  27.       app: kafka-broker
  28.   template:
  29.     metadata:
  30.       labels:
  31.         app: kafka-broker
  32.     spec:
  33.       hostname: kafka-broker
  34.       containers:
  35.         - image: bitnami/kafka
  36.           imagePullPolicy: IfNotPresent
  37.           name: kafka-broker
  38.           ports:
  39.             - containerPort: 9092
  40.           env:
  41.             - name: ALLOW_PLAINTEXT_LISTENER
  42.               value: "yes"
  43.             - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
  44.               value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
  45.             - name: KAFKA_INTER_BROKER_LISTENER_NAME
  46.               value: INTERNAL
  47.             - name: KAFKA_BROKER_ID
  48.               value: "1"
  49.             - name: KAFKA_ZOOKEEPER_CONNECT
  50.               value: "zookeeper-service:2181"
  51.             - name: KAFKA_LISTENERS
  52.               value: INTERNAL://:9092,EXTERNAL://localhost:9093
  53.             - name: KAFKA_ADVERTISED_LISTENERS
  54.               value: INTERNAL://kafka-service:9092,EXTERNAL://localhost:9093
  55.             - name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
  56.               value: "true"
  57.             - name: KAFKA_CREATE_TOPICS
  58.               value: "PARSEREVENT:1:1"
  59.             - name: MY_POD_IP
  60.               valueFrom:
  61.                 fieldRef:
  62.                   fieldPath: status.podIP

我的应用程序成功配置如下：

  1. ...
  2.     consumer:
  3.       bootstrap-servers: kafka-service:9092
  4. ...

而我的offset explorer失败配置如下：

为什么外部连接到Kafka不起作用？

我尝试使用端口转发：

  1. kubectl port-forward kafka-broker-5787578d7f-bmgc5 9093:9093

以及：

  1. kubectl port-forward kafka-broker-5787578d7f-bmgc5 9092:9092

我还尝试在高级选项卡中设置localhost:9092和localhost:9093，并尝试设置空值。所有这些情况都引发了连接超时错误。我应该尝试什么其他方法来解决这个问题？

英文:

I created kafka service and kafka pod in k8s cluster and deployed my application.
So, I can read\write messages to kafka from my application in the same cluster k8s, but I can't connect to kafka using external tools like kafka-tool or offset-explorer
This is my yml for kafka pod and service:

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4. labels:
  5. app: kafka-service
  6. name: kafka-service
  7. spec:
  8. type: NodePort
  9. selector:
  10. app: kafka-broker
  11. ports:
  12. - name: kafka-port
  13. port: 9092
  14. targetPort: 9092
  15. nodePort: 30126
  16. ---
  17. apiVersion: apps/v1
  18. kind: Deployment
  19. metadata:
  20. labels:
  21. app: kafka-broker
  22. name: kafka-broker
  23. spec:
  24. replicas: 1
  25. selector:
  26. matchLabels:
  27. app: kafka-broker
  28. template:
  29. metadata:
  30. labels:
  31. app: kafka-broker
  32. spec:
  33. hostname: kafka-broker
  34. containers:
  35. - image: bitnami/kafka
  36. imagePullPolicy: IfNotPresent
  37. name: kafka-broker
  38. ports:
  39. - containerPort: 9092
  40. env:
  41. - name: ALLOW_PLAINTEXT_LISTENER
  42. value: "yes"
  43. - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
  44. value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
  45. - name: KAFKA_INTER_BROKER_LISTENER_NAME
  46. value: INTERNAL
  47. - name: KAFKA_BROKER_ID
  48. value: "1"
  49. - name: KAFKA_ZOOKEEPER_CONNECT
  50. value: "zookeeper-service:2181"
  51. - name: KAFKA_LISTENERS
  52. value: INTERNAL://:9092,EXTERNAL://localhost:9093
  53. - name: KAFKA_ADVERTISED_LISTENERS
  54. value: INTERNAL://kafka-service:9092,EXTERNAL://localhost:9093
  55. # Creates a topic with one partition and one replica.
  56. - name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
  57. value: "true"
  58. - name: KAFKA_CREATE_TOPICS
  59. value: "PARSEREVENT:1:1"
  60. - name: MY_POD_IP
  61. valueFrom:
  62. fieldRef:
  63. fieldPath: status.podIP

My application success config is

  1. ...
  2. consumer:
  3. bootstrap-servers: kafka-service:9092
  4. ...

And my failed config for offset explorer like this

为什么外部连接到Kafka不起作用？

I tried to user port-forwarding to:

  1. kubectl port-forward kafka-broker-5787578d7f-bmgc5 9093:9093

and like this

  1. kubectl port-forward kafka-broker-5787578d7f-bmgc5 9092:9092

And I tried to set localhost:9092 and localhost:9093 in advinced tab, and tried to set empty value too
All of this cases throws connection timeout errors

What can I try something else for fix issue?

答案1

得分: 2

端口9093仅允许来自Kafka容器内部的连接。这是由KAFKA_LISTENERS设置的；不允许外部连接，尽管您已经命名了协议。

您已经设置了nodePort: 30126，这意味着您需要在KAFKA_ADVERTISED_LISTENERS中定义此端口。然后，您应该能够连接到localhost:30126，而无需进行任何端口转发。

另外，请如我之前提供的链接所述，阅读https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/ 的每个部分（并使用Strimzi，或者使用另一个Operator / Helm Charts，请不要尝试创建自己的Deployment和Service文件）。

另外，Kafka正在移除Zookeeper，因此我建议您使用不需要Zookeeper详细信息的不同工具。这些工具包括基于HTTP的工具，如AKHQ，您可以在Kubernetes中运行并访问，无需为Kafka本身进行端口转发。

英文:

Port 9093 will only allow connections from internal to the Kafka container. This is what KAFKA_LISTENERS sets; not external connections, despite what you've named the protocol.

You've set nodePort: 30126, which means this is the port you need to define in KAFKA_ADVERTISED_LISTENERS. Then you should be able to connect to localhost:30126, and not need to port forward anything.

Otherwise, as I already linked before, please read each part of https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/

(and do use Strimzi, or another Operator / Helm Charts, Please don't try to create your own Deployment and Service files)

Also, Zookeeper is being removed from Kafka, so I suggest you use a different tool that doesn't need Zookeeper details. These include HTTP-based ones like AKHQ, which you can run and access in Kubernetes without port forwarding Kafka itself.

huangapple
  • 本文由 发表于 2023年6月13日 05:50:05
  • 转载请务必保留本文链接：https://go.coder-hub.com/76460529.html
  • apache-kafka
  • kubernetes
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定