英文:
Why doesn't work external connection to Kafka?
问题
我在K8s集群中创建了Kafka服务和Kafka Pod,并部署了我的应用程序。因此,我可以从同一K8s集群中的应用程序读/写Kafka消息,但无法使用Kafka-tool或offset-explorer等外部工具连接到Kafka。以下是我的Kafka Pod和Service的YAML配置:
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
selector:
app: kafka-broker
ports:
- name: kafka-port
port: 9092
targetPort: 9092
nodePort: 30126
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
hostname: kafka-broker
containers:
- image: bitnami/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: INTERNAL
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper-service:2181"
- name: KAFKA_LISTENERS
value: INTERNAL://:9092,EXTERNAL://localhost:9093
- name: KAFKA_ADVERTISED_LISTENERS
value: INTERNAL://kafka-service:9092,EXTERNAL://localhost:9093
- name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
value: "true"
- name: KAFKA_CREATE_TOPICS
value: "PARSEREVENT:1:1"
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
我的应用程序成功配置如下:
...
consumer:
bootstrap-servers: kafka-service:9092
...
而我的offset explorer失败配置如下:
我尝试使用端口转发:
kubectl port-forward kafka-broker-5787578d7f-bmgc5 9093:9093
以及:
kubectl port-forward kafka-broker-5787578d7f-bmgc5 9092:9092
我还尝试在高级选项卡中设置localhost:9092和localhost:9093,并尝试设置空值。所有这些情况都引发了连接超时错误。我应该尝试什么其他方法来解决这个问题?
英文:
I created kafka service and kafka pod in k8s cluster and deployed my application.
So, I can read\write messages to kafka from my application in the same cluster k8s, but I can't connect to kafka using external tools like kafka-tool or offset-explorer
This is my yml for kafka pod and service:
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
selector:
app: kafka-broker
ports:
- name: kafka-port
port: 9092
targetPort: 9092
nodePort: 30126
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
hostname: kafka-broker
containers:
- image: bitnami/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: INTERNAL
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper-service:2181"
- name: KAFKA_LISTENERS
value: INTERNAL://:9092,EXTERNAL://localhost:9093
- name: KAFKA_ADVERTISED_LISTENERS
value: INTERNAL://kafka-service:9092,EXTERNAL://localhost:9093
# Creates a topic with one partition and one replica.
- name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE
value: "true"
- name: KAFKA_CREATE_TOPICS
value: "PARSEREVENT:1:1"
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
My application success config is
...
consumer:
bootstrap-servers: kafka-service:9092
...
And my failed config for offset explorer like this
I tried to user port-forwarding to:
kubectl port-forward kafka-broker-5787578d7f-bmgc5 9093:9093
and like this
kubectl port-forward kafka-broker-5787578d7f-bmgc5 9092:9092
And I tried to set localhost:9092 and localhost:9093 in advinced tab, and tried to set empty value too
All of this cases throws connection timeout errors
What can I try something else for fix issue?
答案1
得分: 2
端口9093仅允许来自Kafka容器内部的连接。这是由KAFKA_LISTENERS
设置的;不允许外部连接,尽管您已经命名了协议。
您已经设置了nodePort: 30126
,这意味着您需要在KAFKA_ADVERTISED_LISTENERS
中定义此端口。然后,您应该能够连接到localhost:30126
,而无需进行任何端口转发。
另外,请如我之前提供的链接所述,阅读https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/ 的每个部分(并使用Strimzi,或者使用另一个Operator / Helm Charts,请不要尝试创建自己的Deployment和Service文件)。
另外,Kafka正在移除Zookeeper,因此我建议您使用不需要Zookeeper详细信息的不同工具。这些工具包括基于HTTP的工具,如AKHQ,您可以在Kubernetes中运行并访问,无需为Kafka本身进行端口转发。
英文:
Port 9093 will only allow connections from internal to the Kafka container. This is what KAFKA_LISTENERS
sets; not external connections, despite what you've named the protocol.
You've set nodePort: 30126
, which means this is the port you need to define in KAFKA_ADVERTISED_LISTENERS
. Then you should be able to connect to localhost:30126
, and not need to port forward anything.
Otherwise, as I already linked before, please read each part of https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/
(and do use Strimzi, or another Operator / Helm Charts, Please don't try to create your own Deployment and Service files)
Also, Zookeeper is being removed from Kafka, so I suggest you use a different tool that doesn't need Zookeeper details. These include HTTP-based ones like AKHQ, which you can run and access in Kubernetes without port forwarding Kafka itself.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论