为什么外部连接到Kafka不起作用?

huangapple go评论144阅读模式
英文:

Why doesn't work external connection to Kafka?

问题

我在K8s集群中创建了Kafka服务和Kafka Pod,并部署了我的应用程序。因此,我可以从同一K8s集群中的应用程序读/写Kafka消息,但无法使用Kafka-tool或offset-explorer等外部工具连接到Kafka。以下是我的Kafka Pod和Service的YAML配置:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  selector:
    app: kafka-broker
  ports:
    - name: kafka-port
      port: 9092
      targetPort: 9092
      nodePort: 30126
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kafka-broker
  name: kafka-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      hostname: kafka-broker
      containers:
        - image: bitnami/kafka
          imagePullPolicy: IfNotPresent
          name: kafka-broker
          ports:
            - containerPort: 9092
          env:
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"
            - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
              value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
            - name: KAFKA_INTER_BROKER_LISTENER_NAME
              value: INTERNAL
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper-service:2181"
            - name: KAFKA_LISTENERS
              value: INTERNAL://:9092,EXTERNAL://localhost:9093
            - name: KAFKA_ADVERTISED_LISTENERS
              value: INTERNAL://kafka-service:9092,EXTERNAL://localhost:9093
            - name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
              value: "true"
            - name: KAFKA_CREATE_TOPICS
              value: "PARSEREVENT:1:1"
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

我的应用程序成功配置如下:

...
    consumer:
      bootstrap-servers: kafka-service:9092
...

而我的offset explorer失败配置如下:

为什么外部连接到Kafka不起作用?

我尝试使用端口转发:

kubectl port-forward kafka-broker-5787578d7f-bmgc5 9093:9093

以及:

kubectl port-forward kafka-broker-5787578d7f-bmgc5 9092:9092

我还尝试在高级选项卡中设置localhost:9092和localhost:9093,并尝试设置空值。所有这些情况都引发了连接超时错误。我应该尝试什么其他方法来解决这个问题?

英文:

I created kafka service and kafka pod in k8s cluster and deployed my application.
So, I can read\write messages to kafka from my application in the same cluster k8s, but I can't connect to kafka using external tools like kafka-tool or offset-explorer
This is my yml for kafka pod and service:

apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
selector:
app: kafka-broker
ports:
- name: kafka-port
port: 9092
targetPort: 9092
nodePort: 30126
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
hostname: kafka-broker
containers:
- image: bitnami/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: INTERNAL
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper-service:2181"
- name: KAFKA_LISTENERS
value: INTERNAL://:9092,EXTERNAL://localhost:9093
- name: KAFKA_ADVERTISED_LISTENERS
value: INTERNAL://kafka-service:9092,EXTERNAL://localhost:9093
# Creates a topic with one partition and one replica.
- name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
value: "true"
- name: KAFKA_CREATE_TOPICS
value: "PARSEREVENT:1:1"
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP

My application success config is

...
consumer:
bootstrap-servers: kafka-service:9092
...

And my failed config for offset explorer like this

为什么外部连接到Kafka不起作用?

I tried to user port-forwarding to:

kubectl port-forward kafka-broker-5787578d7f-bmgc5 9093:9093

and like this

kubectl port-forward kafka-broker-5787578d7f-bmgc5 9092:9092

And I tried to set localhost:9092 and localhost:9093 in advinced tab, and tried to set empty value too
All of this cases throws connection timeout errors

What can I try something else for fix issue?

答案1

得分: 2

端口9093仅允许来自Kafka容器内部的连接。这是由KAFKA_LISTENERS设置的;不允许外部连接,尽管您已经命名了协议。

您已经设置了nodePort: 30126,这意味着您需要在KAFKA_ADVERTISED_LISTENERS中定义此端口。然后,您应该能够连接到localhost:30126,而无需进行任何端口转发。

另外,请如我之前提供的链接所述,阅读https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/ 的每个部分(并使用Strimzi,或者使用另一个Operator / Helm Charts,请不要尝试创建自己的Deployment和Service文件)。

另外,Kafka正在移除Zookeeper,因此我建议您使用不需要Zookeeper详细信息的不同工具。这些工具包括基于HTTP的工具,如AKHQ,您可以在Kubernetes中运行并访问,无需为Kafka本身进行端口转发。

英文:

Port 9093 will only allow connections from internal to the Kafka container. This is what KAFKA_LISTENERS sets; not external connections, despite what you've named the protocol.

You've set nodePort: 30126, which means this is the port you need to define in KAFKA_ADVERTISED_LISTENERS. Then you should be able to connect to localhost:30126, and not need to port forward anything.

Otherwise, as I already linked before, please read each part of https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/

(and do use Strimzi, or another Operator / Helm Charts, Please don't try to create your own Deployment and Service files)


Also, Zookeeper is being removed from Kafka, so I suggest you use a different tool that doesn't need Zookeeper details. These include HTTP-based ones like AKHQ, which you can run and access in Kubernetes without port forwarding Kafka itself.

huangapple
  • 本文由 发表于 2023年6月13日 05:50:05
  • 转载请务必保留本文链接:https://go.coder-hub.com/76460529.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定