无法创建Kafka主题或连接启用TLS的AWS MSK Kafka代理端点:等待节点分配超时

huangapple go评论77阅读模式
英文:

Unable to create kafka topics or connect to TLS enabled AWS MSK kafka broker endpoint: Timed out waiting for a node assignment

问题

我想要将AWS MSK Kafka集群启用TLS并且想要从我的本地Windows机器连接到经纪人。

我在CloudFormation中使用以下TLS配置创建了MSK集群:

EncryptionInfo:
  EncryptionInTransit:
    ClientBroker: TLS
    InCluster: True

我按照AWS博客中的指南在服务器和客户端机器上启用TLS。
https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption.html

我将本地Windows的Java信任存储(cacerts)复制到了E:\kafka_2.13-3.2.1\tmp\(执行以下命令):

copy "C:\Program Files (x86)\Java\jre-1.8\lib\security\cacerts" "E:\kafka_2.13-3.2.1\tmp\kafka.client.truststore.jks"

并创建了一个包含以下内容的config.properties文件:

security.protocol=SSL
ssl.truststore.location=C:\\Program Files (x86)\\Java\\jre-1.8\\lib\\security\\cacerts
ssl.endpoint.identification.algorithm=https
ssl.truststore.password=changeit

我执行了以下命令来在安全的Kafka经纪人上创建主题:

kafka-topics.bat --create --bootstrap-server  b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic

但是遇到了以下错误:

E:\kafka_2.13-3.2.1\bin\windows>kafka-topics.bat --create --bootstrap-server  b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/E:/kafka_2.13-3.2.1/libs/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/E:/Edifecs/TM/sdk/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
[2023-07-18 00:03:52,275] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (b-2.kafkatlsmskcluster.1lyy3o.c14.kafka.us-west-2.amazonaws.com/10.151.46.36:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2023-07-18 00:04:14,090] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
 (kafka.admin.TopicCommand$)

我还在相关的安全组入站规则中允许了TLS端口9094。我错过了什么?它应该可以使用Java的信任存储,对吗?

英文:

I want to have aws msk kafka cluster to be TLS enabled and want to connect to broker from my local windows machine.

I created MSK cluster using following TLS config in cloudformation:

  EncryptionInfo:
    EncryptionInTransit:
      ClientBroker: TLS
      InCluster: True

I followed this aws blog to enable TLS at server and client machine.
https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption.html

I did copied my local window's java's truststore cacerts to E:\kafka_2.13-3.2.1\tmp\ (executed following command)

copy "C:\Program Files (x86)\Java\jre-1.8\lib\security\cacerts" "E:\kafka_2.13-3.2.1\tmp\kafka.client.truststore.jks"

And created config.properties with content

security.protocol=SSL
ssl.truststore.location=C:\\Program Files (x86)\\Java\\jre-1.8\\lib\\security\\cacerts
ssl.endpoint.identification.algorithm=https
ssl.truststore.password=changeit

I executed following command to create a topic on secured kafka broker

kafka-topics.bat --create --bootstrap-server  b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic

but getting following error

E:\kafka_2.13-3.2.1\bin\windows>kafka-topics.bat --create --bootstrap-server  b-2.xxx.amazonaws.com:9094,b-1.xxx.amazonaws.com:9094 --command-config client.properties --replication-factor 2 --partitions 1 --topic TLSTestTopic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/E:/kafka_2.13-3.2.1/libs/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/E:/Edifecs/TM/sdk/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
[2023-07-18 00:03:52,275] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (b-2.kafkatlsmskcluster.1lyy3o.c14.kafka.us-west-2.amazonaws.com/10.151.46.36:9094) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2023-07-18 00:04:14,090] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
 (kafka.admin.TopicCommand$)

I have also allowed TLS port 9094 in relavant security group in inbound rule. What I am missing? It should work with Java's trust store, right?

答案1

得分: 0

我通过允许正确的IP地址并设置端口为9094在与AWS MSK集群关联的安全组入站规则中解决了这个问题。

英文:

I was able to resolve it by allowing correct IP with port=9094 in inbound rule of security group associated with AWS MSK Cluster

huangapple
  • 本文由 发表于 2023年7月18日 15:36:27
  • 转载请务必保留本文链接:https://go.coder-hub.com/76710469.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定