英文:
There is error message KafkaAdminClient when try to connect to local kafka broker deployed on local k8s
问题
我正在尝试在本地的Kubernetes上部署Kafka，然后需要通过应用程序连接并使用offset explorer。
因此，我使用kubectl创建了zookeeper的服务和部署，使用了以下yml文件：
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
spec:
type: NodePort
ports:
- name: zookeeper-port
port: 2181
nodePort: 30091
targetPort: 2181
selector:
app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: zookeeper
name: zookeeper
spec:
replicas: 1
selector:
matchLabels:
app: zookeeper
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- image: bitnami/zookeeper
imagePullPolicy: IfNotPresent
name: zookeeper
ports:
- containerPort: 2181
env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: ALLOW_ANONYMOUS_LOGIN
value: "yes"
然后，我使用以下yml文件创建了Kafka的服务和部署：
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
ports:
- name: kafka-port
port: 9092
nodePort: 30092
targetPort: 9092
selector:
app: kafka-broker
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
hostname: kafka-broker
containers:
- image: bitnami/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
env:
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper-service:2181"
- name: KAFKA_LISTENERS
value: PLAINTEXT://localhost:9092
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://localhost:9092
- name: KAFKA_CREATE_TOPICS
value: "bomc:1:1"
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
两个服务和部署都已创建并运行。
此外，我为这些服务创建了Ingress：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /health
pathType: Prefix
backend:
service:
name: health-app-service
port:
number: 80
- path: /actuator
pathType: Prefix
backend:
service:
name: health-app-service
port:
number: 80
- path: /jsonrpc
pathType: Prefix
backend:
service:
name: core-service
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: kafka-service
port:
number: 9092
- path: /
pathType: Prefix
backend:
service:
name: kafka-service
port:
number: 30092
- path: /
pathType: Prefix
backend:
service:
name: kafka-service
port:
name: kafka-port
- path: /
pathType: Prefix
backend:
service:
name: zookeeper-service
port:
name: zookeeper-port
但是，当我尝试使用offset key工具连接到这个Kafka时，出现了连接错误。如果我使用localhost:30092作为引导服务器 - 日志中会出现错误。
让我知道如果你需要更多的帮助。
英文:
I'm trying to deploy kafka on local k8s, then I need to connect to it by application and using offset explorer
so, using kubectl I created zookeeper service and deployment using this yml file
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
spec:
type: NodePort
ports:
- name: zookeeper-port
port: 2181
nodePort: 30091
targetPort: 2181
selector:
app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: zookeeper
name: zookeeper
spec:
replicas: 1
selector:
matchLabels:
app: zookeeper
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- image: bitnami/zookeeper
imagePullPolicy: IfNotPresent
name: zookeeper
ports:
- containerPort: 2181
env:
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
- name: ALLOW_ANONYMOUS_LOGIN
value: "yes"
Then, I created kafka service and deployment using this yml
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
ports:
- name: kafka-port
port: 9092
nodePort: 30092
targetPort: 9092
selector:
app: kafka-broker
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kafka-broker
name: kafka-broker
spec:
replicas: 1
selector:
matchLabels:
app: kafka-broker
template:
metadata:
labels:
app: kafka-broker
spec:
hostname: kafka-broker
containers:
- image: bitnami/kafka
imagePullPolicy: IfNotPresent
name: kafka-broker
ports:
- containerPort: 9092
env:
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper-service:2181"
- name: KAFKA_LISTENERS
value: PLAINTEXT://localhost:9092
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://localhost:9092
# Creates a topic with one partition and one replica.
- name: KAFKA_CREATE_TOPICS
value: "bomc:1:1"
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ALLOW_PLAINTEXT_LISTENER
value: "yes"
And both services and deployment created and running
And I have ingress for this services
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /health
pathType: Prefix
backend:
service:
name: health-app-service
port:
number: 80
- path: /actuator
pathType: Prefix
backend:
service:
name: health-app-service
port:
number: 80
- path: /jsonrpc
pathType: Prefix
backend:
service:
name: core-service
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: kafka-service # Название вашего Kafka-сервиса
port:
number: 9092 # Название порта, используемого для Kafka
- path: /
pathType: Prefix
backend:
service:
name: kafka-service # Название вашего Kafka-сервиса
port:
number: 30092 # Название порта, используемого для Kafka
- path: /
pathType: Prefix
backend:
service:
name: kafka-service # Название вашего Kafka-сервиса
port:
name: kafka-port # Название порта, используемого для Kafka
- path: /
pathType: Prefix
backend:
service:
name: zookeeper-service
port:
name: zookeeper-port
but, when I try to connect to this kafka using offset key tool, there is error connection.
When I use localhost:30092 like a bootstrap server - error with logs:
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - Starting application : Offset Explorer
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - Version : 2.3
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - Built : Jun 30, 2022
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - user.home : C:\Users\Roberto
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - user.dir : C:\Program Files\OffsetExplorer2
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - os.name : Windows 10
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - java.runtime.version : 1.8.0_232-b09
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - max memory=3586 MB
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - available processors=8
12/мар/2023 22:32:46.111 INFO com.kafkatool.ui.MainApp - java.security.auth.login.config=null
12/мар/2023 22:32:46.121 INFO com.kafkatool.common.ExternalDecoderManager - Finding plugins in directory C:\Program Files\OffsetExplorer2\plugins
12/мар/2023 22:32:46.121 INFO com.kafkatool.common.ExternalDecoderManager - Found files in plugin directory, count=1
12/мар/2023 22:32:46.121 INFO com.kafkatool.ui.MainApp - Loading user settings
12/мар/2023 22:32:46.153 INFO com.kafkatool.ui.MainApp - Loading server group settings
12/мар/2023 22:32:46.153 INFO com.kafkatool.ui.MainApp - Loading server connection settings
12/мар/2023 22:32:50.103 INFO org.apache.kafka.clients.admin.AdminClientConfig - AdminClientConfig values:
bootstrap.servers = [localhost:30092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
12/мар/2023 22:32:50.126 DEBUG org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [localhost:30092 (id: -1 rack: null)], partitions = [], controller = null).
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name connections-closed:
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name connections-created:
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name successful-authentication:
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name successful-reauthentication:
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name successful-authentication-no-reauth:
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name failed-authentication:
12/мар/2023 22:32:50.188 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name failed-reauthentication:
12/мар/2023 22:32:50.198 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name reauthentication-latency:
12/мар/2023 22:32:50.199 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name bytes-sent-received:
12/мар/2023 22:32:50.199 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name bytes-sent:
12/мар/2023 22:32:50.199 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name bytes-received:
12/мар/2023 22:32:50.199 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name select-time:
12/мар/2023 22:32:50.199 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name io-time:
12/мар/2023 22:32:50.204 WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'group.id' was supplied but isn't a known config.
12/мар/2023 22:32:50.204 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.4.0
12/мар/2023 22:32:50.204 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 77a89fcf8d7fa018
12/мар/2023 22:32:50.204 INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1678649570204
12/мар/2023 22:32:50.214 DEBUG org.apache.kafka.clients.admin.KafkaAdminClient - [AdminClient clientId=adminclient-1] Kafka admin client initialized
12/мар/2023 22:32:50.215 DEBUG org.apache.kafka.clients.admin.KafkaAdminClient - [AdminClient clientId=adminclient-1] Queueing Call(callName=listNodes, deadlineMs=1678649690215) with a timeout 120000 ms from now.
12/мар/2023 22:32:50.215 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Initiating connection to node localhost:30092 (id: -1 rack: null) using address localhost/127.0.0.1
12/мар/2023 22:32:50.228 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name node--1.bytes-sent
12/мар/2023 22:32:50.230 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name node--1.bytes-received
12/мар/2023 22:32:50.232 DEBUG org.apache.kafka.common.metrics.Metrics - Added sensor with name node--1.latency
12/мар/2023 22:32:50.232 DEBUG org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
12/мар/2023 22:32:50.320 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions.
12/мар/2023 22:32:50.320 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1.
12/мар/2023 22:32:50.376 DEBUG org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-1] Connection with localhost/127.0.0.1 disconnected
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196)
at java.lang.Thread.run(Thread.java:748)
12/мар/2023 22:33:11.787 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Node -1 disconnected.
12/мар/2023 22:33:12.766 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Initiating connection to node localhost:30092 (id: -1 rack: null) using address localhost/127.0.0.1
12/мар/2023 22:33:12.767 DEBUG org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
12/мар/2023 22:33:12.767 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions.
12/мар/2023 22:33:12.767 DEBUG org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1.
12/мар/2023 22:33:12.768 DEBUG org.apache.kafka.common.network.Selector - [AdminClient clientId=adminclient-1] Connection with localhost/127.0.0.1 disconnected
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196)
at java.lang.Thread.run(Thread.java:748)
答案1
得分: 1
看起来你缺少经纪人配置，通常我在运行Kafka时，通过我的Docker Compose公开以下配置：
KAFKA_LISTENERS: 'LISTENER_INTERNAL://kafka:29092,LISTENER_HOST://:9092'
KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka:29092,LISTENER_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
当其他服务尝试在Docker Compose集群内部连接时，它使用以下配置：
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper-kafka:2181
这也在https://www.confluent.io/blog/kafka-listeners-explained/中有解释。希望这有所帮助。
英文:
It seems you have missing broker config, usually I have below config exposed from my docker compose when I run Kafka on my CI
KAFKA_LISTENERS: 'LISTENER_INTERNAL://kafka:29092,LISTENER_HOST://:9092'
KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka:29092,LISTENER_HOST://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
And when some other service tries to internally connect within the docker compose cluster it uses below config
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper-kafka:2181
This is also explained https://www.confluent.io/blog/kafka-listeners-explained/ I hope this helps
答案2
得分: 1
你已经配置了Kafka经纪人的NodePort为30092，因此要连接到它，您应该使用<node-ip>:30092作为您的引导服务器地址。
如果您正在运行Minikube，可以使用minikube ip命令找到节点的IP地址。
(尽管您提到连接到localhost:30181上的Zookeeper成功，所以您也可以尝试使用localhost:30092连接到Kafka。)
此外，在这种情况下，您无需通过kubectl进行端口转发，因为NodePort已经向外部暴露。
要调试与Offset Explorer的连接问题，请在其设置中启用调试日志记录，尝试连接到您的经纪人，并检查工具的调试日志以获取错误的详细信息。如果您在Windows上运行，日志应该位于工具可执行文件所在的同一目录中。
英文:
You configured the NodePort of 30092 for your kafka broker so to be able to connect to it you should use <node-ip>:30092 as your bootstrap server address.
If you're running minikube, you can find the node ip with the minikube ip command.
(Although you mentioned that the connection to your zookeeper at localhost:30181 was successful so you might as well try localhost:30092 for your kafka.)
Also, you don't need to port forward via kubectl in this case since NodePort is exposed to the outside world anyway.
To debug the connection problems with Offset Explorer, enable debug logging in its settings, make an attempt to connect to your broker and check the tool's debug logs for the details of the error. The logs should be located in the same directory as the tool executable if you're running on Windows.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论