英文:
Unable to create kafka topics or connect to TLS enabled AWS MSK kafka broker endpoint: Timed out waiting for a node assignment
问题
我想要将AWS MSK Kafka集群启用TLS并且想要从我的本地Windows机器连接到经纪人。
我在CloudFormation中使用以下TLS配置创建了MSK集群:
EncryptionInfo:
EncryptionInTransit:
ClientBroker: TLS
InCluster: True
我按照AWS博客中的指南在服务器和客户端机器上启用TLS。
https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption.html
我将本地Windows的Java信任存储(cacerts)复制到了E:\kafka_2.13-3.2.1\tmp\(执行以下命令):
copy "C:\Program Files (x86)\Java\jre-1.8\lib\security\cacerts" "E:\kafka_2.13-3.2.1\tmp\kafka.client.truststore.jks"
并创建了一个包含以下内容的config.properties文件:
security.protocol=SSL
ssl.truststore.location=C:\\Program Files (x86)\\Java\\jre-1.8\\lib\\security\\cacerts
ssl.endpoint.identification.algorithm=https
ssl.truststore.password=changeit
我执行了以下命令来在安全的Kafka经纪人上创建主题:
kafka-topics.bat --create --bootstrap-server b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic
但是遇到了以下错误:
E:\kafka_2.13-3.2.1\bin\windows>kafka-topics.bat --create --bootstrap-server b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/E:/kafka_2.13-3.2.1/libs/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/E:/Edifecs/TM/sdk/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
[2023-07-18 00:03:52,275] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (b-2.kafkatlsmskcluster.1lyy3o.c14.kafka.us-west-2.amazonaws.com/10.151.46.36:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2023-07-18 00:04:14,090] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
(kafka.admin.TopicCommand$)
我还在相关的安全组入站规则中允许了TLS端口9094。我错过了什么?它应该可以使用Java的信任存储,对吗?
英文:
I want to have aws msk kafka cluster to be TLS enabled and want to connect to broker from my local windows machine.
I created MSK cluster using following TLS config in cloudformation:
EncryptionInfo:
EncryptionInTransit:
ClientBroker: TLS
InCluster: True
I followed this aws blog to enable TLS at server and client machine.
https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption.html
I did copied my local window's java's truststore cacerts to E:\kafka_2.13-3.2.1\tmp\ (executed following command)
copy "C:\Program Files (x86)\Java\jre-1.8\lib\security\cacerts" "E:\kafka_2.13-3.2.1\tmp\kafka.client.truststore.jks"
And created config.properties with content
security.protocol=SSL
ssl.truststore.location=C:\\Program Files (x86)\\Java\\jre-1.8\\lib\\security\\cacerts
ssl.endpoint.identification.algorithm=https
ssl.truststore.password=changeit
I executed following command to create a topic on secured kafka broker
kafka-topics.bat --create --bootstrap-server b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic
but getting following error
E:\kafka_2.13-3.2.1\bin\windows>kafka-topics.bat --create --bootstrap-server b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/E:/kafka_2.13-3.2.1/libs/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/E:/Edifecs/TM/sdk/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
[2023-07-18 00:03:52,275] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (b-2.kafkatlsmskcluster.1lyy3o.c14.kafka.us-west-2.amazonaws.com/10.151.46.36:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2023-07-18 00:04:14,090] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
(kafka.admin.TopicCommand$)
I have also allowed TLS port 9094 in relavant security group in inbound rule. What I am missing? It should work with Java's trust store, right?
答案1
得分: 0
我通过允许正确的IP地址并设置端口为9094在与AWS MSK集群关联的安全组入站规则中解决了这个问题。
英文:
I was able to resolve it by allowing correct IP with port=9094 in inbound rule of security group associated with AWS MSK Cluster
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论